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ABSTRACT ■"•n 

Research was conducted to identify criteria for 
successful research and development innovations in vocational 
education. Sixteen research and development (R&D) impact criteria 
were identified through a search of three data bases. The central 
data base consisted of impact studies of 28 selected state-developed 
and National Center for Research in Vocational Education products, 
conducted between 1978 and 1982. The second data base was a 
computerized ERIC search of 267 impact studies of educational 
innovations. The third data base was seminal theoretical works and 
research studies from innovation diffusion literature. After criteria 
were identified, reviewed, and revised, they were, cla'ssif led within 
one of five R&D program improvement stages: development, 
dissemination, implementation, utilization, and effects. Impact 
criteria were defined for each stage, and salient features of each 
criterion were illustrated by a case example from an impact study. 
Guidelines and caveats for using each criterion were recommended. It 
was concluded that impact potential can be built into R&D products if 
specific criteria and guidelines are met throughout the R&D process. 
(KC) 
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FOREWORD 

The impact of research and development on Vocational 
v education programs is an irpolrtant and pervasive topic- Research 

A 

administrators need to know the impact of R&D both for account- 
ability and for orooram planninq purposes- This report contains . 
criteria for determining the impact of research and development. 
Roth formative and sumrative inpact criteria are proposed for^ 
assessinci potential and actual impact- The criteria are based on 
impact studies of twenty-eight products conducted by the National 
Center for Research in Vocational education and a review of the 
literature. Salient features of each criterion are illustrated 
with excerpts from one or more of the impact* studies. Guidelines 

''and caveats are noted for each of the sixteen criteria. Criteria 
were classified by five R&D program % improvement stages. 
Approaches to assessina impact are suggested. 

Appreciation is extended to the reviewers oFTFiis report, 
Thomas Owens, Northwest Feqional Educational Laboratory; Peter- 
Seidman', Illinois State Board of Education; .Wesley Budke ; 
National Center for Research in Vocational Education; Floyd 
McKinney, National Center for Research in Vocational Education; 
and r:.L. M .cCci*l in ; *'lJrftional Center" fotf Research in Vocational 
Education. A special note o.f thanks goes to the authors, Kay 

- Adams, Debra Bragg, and William Hull, for their development of 
♦criteria for systematic assessment of impact. Recognition is due 
to Constance Eaddis and -Sharon L. Fain for editing and to Jeani 
Gray for typino this report. 

Robert E. Taylor 

Executive Director 

National Center for* Research 

. .i' in Vocational Education * 
q viz 
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EXECUTIVE SUMMARY 



Since 1970, the vocational education community has spent 
over $350 million to conduct research and development (R&D). In 
light of such an investment/ ^research administrators need to know 
not only what changes can be attributed to the use of R&D 
products, but also the characteristics of successful R&D, 

This report was written to help research administrators and 
evaluatoss increase the impact of R&D on vocational education - 
programs. By identifying criteria for successful innovations, 
vocational educators can become more systematic in assuring that 
R&D innovations make a difference* 

Sixteen R&D impact criteria are discussed in this report. 
r ihe criteria were derived from three data bases. The central 
data bkse was impact studies of twenty-eight selected state- 
developed and National Center products. These impact studies 
were conducted by the National Center for Research in Vocational 
Education between 1978 and 1982; botb qualitative and quanti- 
tative data were collected. The second data base was a 
computerized ERIC search of 267 impact studies of educational * 
innovations. The. third . da ta base was seminal theoretical works 
and research studies from the innovation diffusion literature. 
After tentative criteria were identified, they were reviewed by 
participants in the Fifth Nationwide Vocational Education 
Dissemination and Utilization Conference, and revised. 

Each, criterion is classified within one of five R&D program 
improvement stages. The stages are development, dissemination, 
implementation, utilization, and effects. These stages are 
viewed as basically linear ^although in practice work flows in a 
cyclical direction between the stages. Most models for assessing 
R&D impact focus only on the last stage, effects. However the 
"impact potential" of an innovation is determined by activities 
in the earlier stages. Consequently this report discusses both 
formative and summative impact criteria. These criteria should 
be met by an innovation as it flows through each of the R&D-based 
program improvement stages. Impact potential of a product 
accrues at the formative stages. Summative impact is the actual 
effects of an innovation on individuals, organizations, and 
society. 

Impact criteria are defined for each stage-, and salient 
features of eacft criterion are illustrated *by a case example from 
an impact study. Guidelines and caveats for using each criterion 
are recommended. A brie*f\sta tement on each stage, with each 
criterion underlined, follows: 

> 

• Development Impact begins with the development of a 

high quality producj:. Involvement of relevant 
audiences should be used to construct a 
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implement . ( 



user-oriented product. A ^systematically . i 
developed product should be research based, J 
•tested, and revised. 

Dissemination Strategic dissemination to encourage the, 

spread of the R&D product to primary 
audiences is essential. Normally, multiple 
^ communication - chanftels are used to increase 
,\, the >likelihood~ of widespread dissemination . 

''Implementation *A prerequisite for impact is selective 

implementation . Psychological , adminis- 
trative,, and physical support systems are* 
necessary to make an innovation opetational 
Cost .feasibility * studies are helpful in ;* 
deciding how much' c the innovation will cost to 

• Util ization * Integrated use results in integrating a 

product into per-sonal and organizational 
* routines. Impact measures should ailqw for** 

- te multiple patterns ' of use , some of which may 

be unanticipated. Time On task in using* the- 
, innovation enhar^es impact . ^ 

• Effects User satisfaction is a prerequisite to otlier , 

effects. Product use should oresult in 

indiv idual grov/th, organizational change , and 

sbcietal contributions . 
_ « 

Improving vocational education with' R&D products should be a 
systematic, incremental process. Impact potential can be built' . 
in**.o R&D products if specific criteria artd guidelines are met 
throughout the R&D process;. This report should help research 
administrators, project directors and. evaluatofs gain increased 
impact from limited R&D dollars. 

/ 
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CHAPTER I 
' INTRODUCTION 

The -use of scientific evidence to improve programs has been 

part of the vocational education system since its beginning. 

Funds authorized for program improvement in the - Vocational 

Education Act/ of 1963 and' the Vocational Education Amendments of 

*> 

1976 have resulted in numerous research and development (R&D) ^ 
projects* Since 1970, sustained funding of vocational education 
R&D has resulted iiv over $350 million in Expenditures (Hull 
1980). iBudke (1982) reports that between 1-978-82 nearly $105 
million were spent by states on nearly 4,000 program- improvement 
projects. These projects have produced research findings, 
products, ancV graining used to upgrade the knowledge, attitudes, > 
and skills of Isoth vocational educators .and students. Through 
these efforts, a technology of how to conduct research, dis- 

• - • / 

seminate products, and use tfrem to improve people- arid programs 

* • 
has gradually evolved. This technology has not always worked 

0 " " N . > . / * - r ^ 

well. Becabs^ N of the 'lack of coordination between the efforts^ a 

s • - . < , 

mediocre idea is often used * extensively while one with potentiai 
for significant improvement .may be overlooked. The persons most 
likely to use the R&D impact criteria discussed in this, report 

/ . - 1 

are research administrators, e^*aluators\and ochers concerned with 
tire -use of K<xD products to improve vocational educatipn programs. 
Project directors can also use these criteria t*o Monitor 
-development and dissemination activities. 
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There are several terms user! in this report that *need 

a v 
definition, Chief anona then is imp act . In.pact is defined as a 

« ~ ■ 1 — ■— ■ . 

measurable chanae resu 1 tinq I ran an innovation. An innovation, 
can effect actual chanqe through itr. use, and it can acquire the 
potential for change by meeting specific criteria.' Potential for 
chanae is an important dimension of impact assessment because 'it 
is. often impractical to "wait until after an innovation has boon 
completed to assess impact." Funding sources, commitment, to the 
innovation"/ and the need for data are much stronqer during the 
formative stages of the R&D process. Consequently, this report- 
discusses both formative and s ammat ive impact. Formative impact 
can be measured by performance standards that an innovation 
should meet at each formative %taoe of the, R&D program 

improvement process — development , d insemination, implementat ion / 

* 

and use. Summative impact i*s the actual effects of an innovation 
on individuals, organizations, and society. Figure 1 illustrates 
the relationship between formative and summative impact. As 
shown in the f inure, some actual ^-impact can occur during 
formative R&P proaram improvement stages. 

An innovation is defined as an Ldea that is perceived by an 
adoptino unit as a new one. .Innovation ,is used synoporrously with 
R&L product in the context .of this report. ' An R&D product may be 
a research report, an instructional guide, a training' workshop, 
or technical assistance provided by an R&P specialist at the 
roauest of a practicing <=>ducalor* 

• * 

V 

2 

1 O 
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Figure 1. RELATIONSHIP BETWEEN FORMATIVE AND SUMMATIVE IMPACT 
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Other terns used in this report are defined as follows: 

Adopting unit — An individual or group of* individuals who 
use an* innovation. 

Primary user — The person or persons for whom the product 
,^as developed. 

Secondary user — A person or persons who use the product as a 

direct result of the primary user 1 s influence . 



Purpose of Objectives 

The purpose,, of this report is to expand the reader's concept 
of impact stemming from the development and use of. research-based 
products. It encompasses not only the usual concept of impact 
(as an effect of using products), but also includes preconditions 
to the development of impact for research and development 
activities. Additionally formative impact may occur during the 
earlier staqes of the R&P program improvement process. Fxamples 
of "how an P&D product may make things different" in each stage 
of the program improvement process are illustrated below using 
A the results of the Performance-Based Teacher Education ( PBT£ ) 
modules developed at the National Center for Research in 
Vocational Education (National Center). These staqes are 
discussed in Chapter II. 

Stage Example of Impact 

• Development Draft versions of PBTE modules were tested in 

eighteen field sites prior to 1977. Un- 
doubtedly, the teachers and students usinq 
these modules were affected either positively 
or negatively as a result of this use even 
before the modules were placed on the market* 
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Oyer ?f0, 000 ror^nJes were dissenunat ed between 
t'arch 1°77 and January l^PO. Recipients' 
awareness of the modules vas rhanaer 1 as a 
result of th is act ivi ty. 

m eppie University professors allowed students 
to use the PPTF modules in any sequence and at 
thei r own pace, thereby alter ina the us ual 
structure of this i nst itut ion 1 s teacher edu- 
oa t ion course work. 

^be complet i on of fc'tv-f ive PPTF modul es was 
subst i t ut ed -For th i rtv-s ix credi t s in the 
t eacher certification procram at the Pniver- 
<;itv o-f Central FJorida, displaoinc other 
courses in t eacher rreparat ion. 

Reports from faculty, sta^f, and students 
indicated the tire reauired t<? certify tea- 
chers had been shortened . 

fuoh chances as a result o+ P£P are unioue to a part ioular* staae 

r c ir^rrverert ir that there is no nuaranteo that the impact f rom 

opo sta^e would he passed on to the next. However, product users 

durino an.^ariv staoe of development often become advocates at a- 

later staoe. °ur]' additional advocacy undoubtedly increases the 

irpprf of a rro^nrt at the utilization or effects staaes. 

rT¥ h*> foilov/inrr t-wo objectives were used for this report: 

1. ^o select criteria for determinino the impact of R&P 
within each stace of the pronram improvement process. 

9 . m o illustrate salient features o f each criterion 
with representative cases from impact studies. 

Tmpaot data, either Normative or summative, can be used 

either for accountability or for plannino future P&P inter- . 

vent ions. Tn a sense, impact data can become needs -sensino data 

for future pronre\r&. This contribution of impact data to 

planninc and no 1 i cv formulation is ouite different fror 1 the 
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account ah i J it v motivation that* often drives impact. 



Methodology ■ 

Several procedures were used to identify the impact 
criteria. First, three data bases for deriving impact criteria 
were established. The central data base included impact studies 
of twenty-eight selected R&D products conducted by the National 
Center between 1978 and 1982- The second data base was a 
computerized search of impact studies in the Educational 
Resources Information Center (URIC) . Two hundred sixty-seven ■ 
studies were identified in this data base, many of them 
follow-ups of research-based products* The third- data base 
included selected works from R&D product diffusion literature, 
*After this literature was examined for significant impact 
criteria, a list of twenty-eight criteria were identified . The 
criteria were classified by, stages in the program improvement 

- process . These criteria were then revised by project staff, by 
participants in the Fifth Nationwide Vocational Education 
Dissemination and Utilization Conference, and finally by draft 
report reviewers. Sixteen R&D impact criteria resulted . 

Excerpts were selected from the impact studies of vocational 
education R&D products and programs conducted by the National 
Center between 1978 and 1982. Eight of these were state- 
developed program improvement products and twenty were National 
Center products and programs. These studies examined impact as 
well as analyzed factors which contributed to impact. In most 
cases, the data were collected on site through interviews, 
observation, and reviews of records. Additionally, mail surveys ^ 

- were used to assess the distribution, use, and impact of the 
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state-developed products. Several of the National Center 
products were studied through telephone interviews with 
recipients; these studies provided cumulative evidence of impact 
iron* different types of K&D-based innovations. 

Limitations of the Report 

1. The language and concepts used to portray impact are 

Linked to the k&D process-. All of the studies used to illustrate * 

/ 

the impact criteria were based on utilization of R&D products. 
This Limits interpretation and implications from these f indings ^ . 



to research-based program improvements. 

2. A "product" mentality pervades these criteria., This was 
not deliberate; it was merely convenient, transportability of 

m 

Ri*D results is enhanced by ^packaging the innovative finding or 
procedure in a way that is acceptable to others. However, the 
reauer should keep in mind that most R&D generates many effects 
(such as a changed attitude in people who are associated with it) 
that cannot be packaged into a product; they must be exper- 
ienced. Because of this, training workshops and technical 
assistance efforts are considered R&D products. 

Toward a Conceptual Framework of R&D Impact 

Current limitations . in R&D funds place pressure on 
researchers to ensure program improvement. Accountability has 
become a major concern for both researchers and program managers. 
With this in mind, the authors have examined the program improve- 
ment literature for ways to enhance impact. 
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Three bodies of knowledge provide guidance for identifying 
the critical ingredients of R&D impact. fhese are (1) theory and 
models that define innovation diffusion and knowledge utili- 
zation, (2) large-scale empirical.studi.es of the change process, 
and (3) evaluations of vocational education R&D efforts that 
appear to have had impact. A brief description of each of these 
knowledge bases follows. 

Theory & Models 

Conceptual frameworks for describing the processes of 
.innovation diffusion and knowledge utilization have abounded in 
the literature. Rogers (1962) described a five-stage adoption 
process that involved carefully guided movement from initial 
awareness, interest, evaluation, and trial to final adoption of 
an innovation. Rogers included descriptions of innovation 
characteristics, environmental facilitators and inhibitors, and 
categories of adopters in his ' diffusion model. 

Havelock (1971) added the personal linkage perspective. He 
emphasized the importance of interpersonal linkages among 
researchers and practitioners in problem identif icatiqn and 
solution. Kotler (1972) built a repertoire of change strategies 
to be used in diffusing innovations. Chin and Benne (1969) 
described three categories of strategies: coercive, persuasive, 
and reeducative. Zaltman and Duncan (1977) expanded the realm of 
possible change strategies to include an arnay of tactics ' from 
direct mail to network building. The Dissemination Analysis 
Grou£ (1977) reorganized Rogers model into four levels: spread, 
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exchange, choice, and implementation. Hall and Loucks (1977) 
added the concept of "levels of use" of innovations from 
routine use to renewal. Sieber (1968) expanded the lis£ of 
environmental Situations that affect innovations ~by considering 
factors >such as resistance to innovation and goal conflicts. 
Later, Sieber (1981) added conceptual guidelines for incentives 
and disincentives to knowledge utilization. Louis, Rosenblunt, 
and Molitor (1981) defined Characteristics of external agents 
that facilitate knowledge utilization. Most recently, the 
concept of "system linkage" has been emphasized in diffusion 
.literature. Weick (1976) introduced the concept of "loose- 
coupling" or the establishment* of informal, voluntary, and often 
nonrational linkages within and among organizations^. The 
diffusion literature base provides conceptual frameworks for 
identifying facilitators and inhibitors of R&D impact. 

Within vocational education, the National Center has 
conducted numerous studies that apply diffusion literature. One 
of the most widely adopted was Innovations Evaluation : A 

4 * 

Consumer's Guide (Hull 1*971). The National Center (1979) also 
developed Tentative Product S.election Criteria for the National 
Dissemination and Utilization System for Vocational Education. 
The tentative product selection criteria in this publication were 
I organized ^into five categories: effectiveness, compatability , 

content, cost-efficiency, and research evidence of effectiveness . 
The Illinois State Board of Education (1980) has developed 
detailed guidelines for dissemination and assessing impact of 
vocational education program improvement products. 
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Empirical Studies * 
* 

Several large-scale studies have been conducted to determine 

f 

why research and uevelopment products do or dj5)not have impact: 

Berman and McLaughlin (1978) studied f©ur federal change programs 

including the Lxemplary Programs mandated in the 1960 Amendments 

of the Vocational Lducation Act. A £ey finding from this study 

was the two- fold process of adapting innovations. In this 

♦ 

process, both the product and settincj were changed by imple- , 

l ' 

mentation activities. In 1977, Abt Associates in Cambridge, < 

Massachusetts, began a study of a three-year experiment in the 

delivery of research-based products for school improvement (Louis 

al. 1982). The study found participation by the entire staff 

in^ a problem-solving process to be important. There was no 

f 

relationship found between the costs for school improvement 
efforts and the chances of success. A 

A massive .study of dissemination efforts that support school 
improvement was undertaken by the NETWORK in 1978 (Crandall et 
al. 1982). The primary finding was that the major factor 
producing change in classroom practice was the amount of 
classroom time spent on the new practice. Time on task appears 
. to be an important variable in producing teacher commitment. 
Clearly, progress in understanding impact is being made, but much 
remains to bygone in its measurement. 

Recent refinements of knowledge utilization concepts through 
empirical studies have emphasized lirikage, networking, and loose- 
coupling. Adoption of innovations is now viewed as .a more 
diffuse, interactive, and adaptive process then it was in the 
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earLy 1960s . Thus mutual adaption rather than adoption may 



better characterize the utilization of new knowledge. 
Evaluations of Vocational Education R&D 

— - - .. — — . . — 

Within vocational education, three large scale studies of 
KuD have been conducted: (1) The Committee! of Vocational 
Education Research and Development (COVERD 1976) assessed R&D . 
conducted between 1965 and 1974, (2) Development Associates 
( 1975) studied career .education projects, and (3) the General 
Accounting Office (1974) studied vocational programs funded under 



of R&D impact on students. After an analysis of these studies^ 
Rita (1982) recommends additional theoretical, research, empirical 
studies of the funding process, and the development of compre- 
hensive frameworks for impact studies. * 



the Vocational Act of 1963. 



These -"studies found 
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CHAPTER II 

R&D IMPACT CRITERIA BY 
PROGRAM IMPROVEMENT STAGES ' 

Chances resultinq from the use of JR&D products may be 
pervasive, may be short or long term, and- may be difficult to 
detect. The 'procedure for impact assessment should be carefully 
developed for each program or product'. Time is required for 
impact to. take place, so it is helpful to identify indicators of 
impact for each different staae in the R&D p^bgram improvement 
process . 

This chapter contains sixteen indicators of impact in the 
form of criteria classified according to the five stages in the 
program improvement process. Evaluative information from impact 
studies in itself dor ^ not, assure program improvement. In fact, 
it may indicate very little improvement has taken place. 
HoweJpr, a comparison of observations against*^Jfche criteria 
proposed in this report does provide a basis for estimating the 
likelihood of impact from an R&D product. For example, a product 
disseminated to only one county in a state is likely to have less 
impact (other factors beinq equal,) than a similar product 
disseminated throughout a state. In most case?, of course, 
resources are limited; thus the same number of products spread 
throyohout the state would have to be more strategically 
disseminated than one which is concentrated in a single county. 

The impact criteria in this report are based on 'five years 
of studies at the National Center and a review of related 
literature. The criteria are applicable to research, 
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development, and training, projects. They are listed in -this 
-chapter for easy review by the reader, and are preceded by a 
conceptual framework to help explain the R&D program improvement 
process— ea^ which they are based. The impact criteria are grouped 
by stages in this framework. Each of the five stages in the 
framework are explained prior -to defining the criteria. 

R&D Program Improvement Stages 

The framework in figure 2 places the five stages of program 
improvement in a linear sequence. The framework depicts one of 
several existing models of the program improvement process. The 
chosen framework is logical and illustrative of the flow. of 
product developmeni^*to program improvement « However a linear 
sequence Cdn be criticized for not recognizing the cyclical flow ( , 
of work usually associated with R&D processes. For example, 
products may be significantly redesigned during the imple- 
mentation stage. Additionally, a linear model does not depict 
interactive linkages between users and researchers. Recognition 
of the importance of viewing program . improvement as an 
interactive and cyclical process is represented by the dotted 
feedback lines in the figure* The chosen framework, although 
somewhat static and lock-step in appearance, provides a con- 
yenient and logical frame of reference for identifying impact 
criteria. Each of the stager depicted in figure 2 are discussed 
in the following section. 

The first stage of the process for improving programs is 
development. The term development refers to ,the span of the R&p 
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process from basic research to a completed, tested product or 

c 

, 4 

program- This may occur on site, in ^ local education agency' 
(LEA), or may take' place, som<g distance from local instructional ^ 
activities. R&D funds are used by* local education agencies state . * 
departments , universities , a,nd ^enters or regional laboratories 
to devise Fiew and better educational innovations. The con- , 
centration of expertise in development* at these locations, may be 
efficient, but barriers ^Eo acceptance of the new ideas are 
created by both^-physical distance and by users 1 'lack of 

ownership. Thus, it is critical that the innovation be relevant 

.) . - • y 

to the needs of the potential adoption site. ljjis. is why the , 

• *• 

next stage % of program improvement is so important.- 

Dissemination must be considered in improvement .plans ev.en 

if the innovation is developed at' a local site such as a school' 

district ♦ An innovation should be transported to other locations 

to keep money from being wasted by others "rediscovering the 

wheel." Agencies that sponsor an innovative development often , 

must take on the responsibility of spreading it to other sites. 

A local school has few incentives to disseminate new ideas 

outside of its own district. Dissemination costs for exemplary 

programs/products may be borne by national organizations such as " 

the National Diffusion Network (NDN) or state agencies such as a 

research coordinating unit. (These organizations are interested 

in nationwide and statewide program improvement, respectively.) 

Normally, these agencies provide information about the exemplary 

program and, on occasion, fund sites to demonstrate the program. 

^insemination is .clearly an important stage in the program 
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it iprovoiutMi t [;r< k *«-j»:> . I*oopl«» l^arn of innovations from their 

.friends printed , information , conferences, etc. % * . 

Impleme nta tion bridges the gap between learning about the 

ifsnpvatiion "and actually trying i£*. Rogers (196^) tailed about 

* 

tho trial use of an innovation. His discussion is relevant if 
the adopter is an inctdvid aal . .However, the process of 
incorporating an innovation into* an organization is quite a bit 
uor'e complex."' Various parts of an brganizati jn must be assessed 
and analyzed to determine the effect of the new idea on them. 

Groups of people oft,en must be consulted and -disruption 
minimized in oruer to encourage the 'next stage of R&D-based 
program improvement, utilization. Actual use of .an innovation 
u\ust precede any, claims' of effects.* Use* varies from partial to 
complete, and from anticipated to unanticipated. Sometimes the 
proJuct' or program is discon tinueu ' soon after adoption or is > 
modified as it. is used at the new ?ite. Unique adaptations of an 
innovation on site can lead to creative integrated, beneficial , 
uses. However, commensurate with these adaptation9^should be 
adjustments k\\ the expectations for what the modified innovation 
can achieve. 

The final stage of program improvement is gf fee ts . Measur- 
ing the effects of ari R&D innovation constitutes a search for 
visiole changes as a result of its use. Attributing causality to 
an innovation is extretaely difficult because conditions that 
afiect program outcomes are many and interrelationships among 
variables are complex. Nevertheless, .an impact evalyator must j 
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sort out the'^f actors that influence* program effects and describe 
„them as completely as the data will permit. ' 

R&D Impact Criteria 

Criteria for assessing the impact of R&D products are 

arrayed along five stages in figure 3. Two different types of 

impact criteria are presented: formative and summative. 

, Criteria inherent in the first four stages in the framework 

(development/ disser nation, implementation, and utilization) are 

considered to be formative impact criteria. The primary purpose 

& 

of evaluations conducted during these stages is to enhance the 
impact potential of the innovation. Criteria related to the last 
stage, effects, are considered summative criteria. Changes in 
people,* ornanizations or society are primary indicators of 
summative impact. Evidence on the permanence of the changes and , 
unanticipated conseguences of the innovation sometimes occur 
months or even years after the innovation has been introduced. 

Securing the funds, time, and interest for conducting impact 
assessments is difficult. Consequently, the authors advocate 
formative impact assessments as well as summative impact 
assessments.' Formative impact assessments would examine the 
extent to which important criteria have been met at each stage of 
the R&D process. The effect of an R&D product in meeting these 
criteria is cumulative. For example, a product that is system- 
atically developed, strategically disseminated, selectively 
implemented, and used in an integrated manner is likely to result 
in greater change than is a product not meetino these criteria. 
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of course, the impact of an R&P product also reflects the 

constraints and opportunities present in the adopting unit's 

i 

environment. Thus a> product developer or disseminator does not 
have total control over changes that may or may not take place. 
Definitions of sixteen P&D impact criteria organized by the five 
program improvement stages follow. 

Development 

Impact begins with development of the product. Thus 
criteria used in the development process can subsequently be used 
to select relevant research for developing other products, to 
encourage systematic testing/revision, and to upgrade product 
quality. 

Systeratic development . A systematic process should be 
followed in developing innovations. An ideal process would 
include conducting research/needs assessment/task analysis; 
reviewino relevant knowledge/practice; buildino a conceptual 
framework; sequencing development; conducting testing and 
revision cycles; disseminating the product; implementing the 
product; and evaluating the results. 

High qualify . Innovations should reflect scholarship, be 
useful, communicate clearly, be marketable, and be free of 
biases. Content should be accurate, up-to-date, focused on 
essentials , and complete . 

User orientation . Representat Ives of relevant aud iences 
should be identified and involved in designing, testing, and 
using innovations. Primary audiences should receive priority in 
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«: isscrinat ion efforts. The resulting product should contain 
practical information organized in an \2asy-t0-use format. 
Pisseriination 

The dissemination cr iter ia shou Id encourage the spread of 
the R&D product to primary audiences, increase the likelihood of 
the product's acceptance, and generate support "for its use! 

Strate gic dirsemi nation . Cost-effective strategies for 
disseminating an R&D procluct should be devised based on ^ 
characteristics of potential users; site-specific factors; and 
features of the product itself. Dissemination should be 
strategic in reaching opinion leaders and influential, organi- 
zations in the external environment. 

Multiple channels * More than one channel for conveying 
information about innovations should be used. Communication 
should include mass media (e.g., direct mailing of brochures) and 
interpersonal channels (e.g. technical assistance). Normally, 
information duplication and overlap are assets rather than 
liabilities during the dissemination stage. 

Widespread dissemination . Innovations shou Id reach as many 
potential users as possible. Thus dissemination to individuals 
in different roles, in diverse settings and in many geographic 
areas should be emphasized. Secondary dissemination through 
workshops , reprints , libraries , the ERIC system, and so on shou Id 
be encouraged. 

Implementation 

Implementation strategy determine^; the product 1 s point of 
entry into an organization (e.q., at the classroom level). Cost 
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feasibility studies and the need for support systems, aid the 
timely implementation of R&D. 

Selectiv e implementation . The introduction of innovations 
should be sequenced to meet the needs and unique characteristics 
of an adopting site. A process of mutual adaptation between the 
site and the innovation should be encouraged. 

Support systems . Support systems necessary for encouraging 
the full use of an innovation should be operational at the time 

4 

of implementation. These systems are of three types: personal 
resources (e.g., administrative endorsement, site personnel 
endorsement); information resources (e.g., training in the use of 
support materials and % procedures); and physical resources (e.g., 
dollars, supplies, and equipment). 

Cost feasibility . Information describing the innovation 1 s 
resource requirements should allow quick and easy estimates of 
costs likely to be incurred by an adopting unit. 

Utilization 

Various product use criteria can encourage the appropriate 
trial use of products, stimulate their integration with existing 
operations, and increase the chances of their continued use. 

Multiple p atterns of use. An innovation's use patterns will 

■ £ = " 

vary according to the conditions of use received, its intensity, 
level, frequency, and extent. The users 1 setting, role, and 
demographic characteristics create the conditions for different 
types of use. Multiple patterns of use and secondary use of R&D 
by other than the primary user audience should be encouraged. 




T iv\e on task * An P&n product should be used frequently 
enouqh and long enough for its use to become an integral part of 
current practice* The audience's tine in actually using the 
product should be maximized • 

Integrated use . The use of an innovative product should be 
intensive and pervasive throughout the organization . Accom- 
plishing this task will require drawing o.n personal, cbmmitment 
within the organization to institutionalize the product into 
organizational routines. ^ * f 

Effects 

Product effects criteria should accurately describe changes* 
in individuals, organizations, or society attributed to use of 
R&D innovations. 

User satisfaction . The R&.D product and its implementation 
should meet users' expectations and result in a positive user 
attitude toward the product. User satisfaction may be indicated 
by product advocacy and/or creative adaptions. ^ 

Individual growth . Innovations should contribute to changes 
in an individual's attitude, knowledge, and/or performance. 

Organizational change . R&D products should contribute to 
beneficial changes in the user's organizational policy, programs, 
practices and/or structure. Furthermore, these changes may 
reflect cost and tine savings over current practice. 

Societ al contributions . .R&D products should contribute new 
and siqnificant information with the potential to advance know- 
ledge, improve current practice, and/or influence social systems. 



CHAPTER III 

DISCUSSION OF R&D IMPACT CRITERIA 

This chapter describes each impact criterion in greater 
detail. For each criterion , the following sections are provided: 

• Definition > ^ 

• Overview 

• Illustrative Case ( s ) 

• Guidelines 

• Caveats 

The definitions for each step in the systematic implementation 
process as indicated in Chapter II is repeated, and then expanded 
through an overview of the related research and component parts 
of each criterion. Each criterion is illustrated with one or 
more examples from actual vocational education innovations. 
These illustrative cases are excerpts* front impact studies of 
eight state-developed products and twenty products and programs 
produced by the National Center between 1978 and 1982. Guide- 
lines for using the criteria and for utilizing the caveats on 
problems that may occur are then presented. The guidelines are 
intended to help the reader implement the criteria in their 
particular situation. Guidelines for , the effects stage criteria 
tend to focus on assessing impact. The effects criteria, unlike 
the criteria in the previous four stages, indicate summative 
impact. The caveats identify some of the pitfalls to avoid in 
meeting each criterion. / 
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Systematic Development 



A systematic process should be followed in developing in- 
novations. An ideal process would include conducting 
research/needs assessment/task analysis; reviewing relevant 
knowledge/practice ; building a conceptual framework ; se- 
quencing development ; conducting testing and revision 
cycles ; disseminating the product ; implementing the product ; 
and evaluating the results. 



Overview 

ny models exist for systematic change. Guba and Clark's 
(1965) Research, Development, Diffusion, and Adoption model was 
one of the first calling for programmatic activities. Later, a 
fifth phase (evaluation) was added to this model. Stuf f lebeam* s , 
et al . (1973) CIPP model — Context, Input, Process, Product — 
focused primarily on evaluation as it relates to each phase of 
program development. Rosenberg's (1982) Instructional Systems 
Design (ISD) model includes the phases of analysis, design, 
development, implementation, and evaluation. 

Most models calling for systematic development use a linear 
process. Typically, these models incorporate both evaluative 
feedback and the recyling of feedback,, but they basically move in 
a sequential, somewhat lock-step manner from point A to point B. 
Other authors have developed linkage models tha.t emphasize 
interactive user involvement throughout the research and 
development process. Havelock and Lindquist (1980) place users 
in the center of the linkage model by moving them through the 
stages of identifying and solving user problems. These stages 
are (1) arousal and articulation, (2)' communication, (3) imple- 
mentation, (4) scientific problem-solving, (5) transformation, 
(6) dissemination, (7) utilization, and (8) user problem solving. 

Ideally, the R&D process should be based on scientific 
knowledge and processes as well as on practice. User's ideas, 
input, feedback, and experiences should be systematically 
incorporated at critical points in the R&D process. 



Illustrative Case 

The case of the National Center's Performance-Based. 
Teacher Education (PBTE), a vocational education curriculum, 
series (Hamilton et al. 1977) provides an example of system- 
atically developed R&D (Adaihs, MacKay and Pattbn 1981). This 
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curriculum was developed over a ten-year .period. PBTE was 
* extensively researched, fully tested, and carefully written. 
Work on PBTE began at The National Center in 1967 with a research 
study to determine important competencies for vocational- 
technical teaqfiers (Cotrell, Chase, and Molnar 1972). Following 
identification and verification of 384 competencies, modules were 
developed. These 100 modules, covering the essentials of 
teaching, formed the core of PBTE. Throughout the modules' 
development, emphasis was given to continuous refinement through 
iterative cycles of development, testing, and revision. 
Supporting materials were also developed. In summary, the 
development of PBTE included basic research from 1971 to 
1974; field testing from 1975 to 1976; training personnel for 
implementation from 1976 to 19.78; and dissemination and 
utilization from 1977 to the present. 



Guidelines 

•„ Conduct a needs assessment to ensure a focus on an 
enduring problem without neglecting immediate needs. 

• Review relevant research and practice to ensure that the 
product builds on prior work, is nonduplicative, and 
makes a contribution to knowledge. 

• Base design of the product on appropriate theories, con- 
ceptual frameworks, input from potential users, and 
practices that work. 

• Develop the product through systematic steps. 

• Test and revise the product in phases (e.g., pilot 
testing, field testing, expert review, and user review). 

• Plan for widespread dissemination of the results through 
multiple channels. 

• Provide for assistance to us ! ers during the implementation 
stage. 

» Evaluate the product to assess user satisfaction and its 
effects on individuals, organizations, and society. 



Caveats 

The push-pull between scientific precision and pragmatic 
consideration must be carefully balanced. At times, 
systematically conducted R&D must be approximated during 
certain steps. 



Useful substitutes for standard R&ft procedures include a 
user panel instead of a na^ds assessment, an expert 
consultant or panel instead of a literature review, and 
reviewers instead of testing • These substitutes , used 
effectively, can result iri^igh-quality products that 
meet the needs of recipients ♦ 

If one takes too fnany shortcuts in the scientific process 
or moves too far toward user-oriented concerns, R&D that 
merely reinforces conventional -wisdom , may result. This 
would not advance knowledge* Instead it would waste, 
valuable resources • 
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High Quality 



Innovations should reflect scholarship; usefulness ; com- 
municability ; % marketability; and equity. Content should 
be accurate, up to date, focused on essentials., and complete. 



Overview 

y 

Quality, always somewhat in the eye of the beholder, is a 
nebulous yet essential ingredient. Scholarship may be less 
important to some groups than others. Individuals with different 
values, biases, an# preferences will have different quality 
standards.* Yet quality is an essential ingredient for all 
innovations . 

A product of high quality will tend to be used over time and 
have deeper effects than one of lesser quality. The Education 
Products Information Exchange (Komaski 1978) compiled a synthesis 
of minimum quality criteria for educational products. These 
criteria are especially useful for instructional ^materials. 
Krause and Adams (1982) also developed criteria for ensuring 
quality in research and knowledge products based on. y f ive ' 
categories of criteria: (1) scholarship, (2) usefulness, 
(3) equity, (4) conununicability , and (5) marketability. Specific 
criteria and standards for each of these dimensions of quality 
need to be defined. 



Illustrative Case 

One of the most sought after products distributed by the 
Dissemination and Utilization Program at the National Center was 
the Handbook for Teachers of Adult Occupational Education . This 
easy-to-read and informative digest on adult education teaching 
methods was developed by the Bureau of Occupational Education 
Curriculum Development of the New York State Education Department 
(1977). The handbook assisted individuals in becoming competent 
instructors in their field. It also answers some of the 
questions ne\Vinstruotors have during their first few weeks of 
teaching. The- content include information about students, 
principles -of teaching, testing and evaluation of students, 
curriculum develppoment , planning for instruction, and classroom 
management . . * 

The handbook was practical, straightforward, and oriented to 
adult teachers. It was not an in-depth treatment of teaching 
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methods . Features of the product that appealed to users wer.e the 
following: " " 

» Adaptable; easy-to-use format . Instructors could find 

the information they needed and read t;he sections in a 

few minutes* The illustrations lent themselves to a wide 
p - ' variety of settings. 

» Highly focused content . The information was designed for 
first-time teachers of adults and/or instructors of 
occupational programs. The information was relevant to 
their needs and perceived by some as sufficient for new 

instructors . 
» * 

• Unbiased material . No sex or social biases were 
found in the material. 

m Attractive layout . The booklet was very readable and 
packaaed with illustrations. 

Many factors account for this product's high level of use. 
In addition to the high-quality content, the booklet met a 
pervasive, critical need: the upgrading of skilled craftsmen 
into competent instructors. A support system was already in 
place (e.g., the teacher education staff in colleges and 
universities) to promote the product. Today, the product is 
being used as a primary teacher training resource by a network of 
community colleges in South Dakota. CETA p'rogram instructors, 
local education agency in-service programs, postsecondary teacher 
training programs ^ teacher certification programs, and private 
trade schools are also are using the product. 



G uidelines 

• Ensure the content is accurate, up-to-date, system- 
atically researched/developed, focused on essential 
subject matter, and complete in its references. 

• Present the information in a format that is practical, 
relevant to user needs, complete, adaptable to different 
settings/audiences, and capable of extended utility. 

• Use language that encompasses both sexes, avoids stereo- 
types, and" represents different special need groups 
through examples. 

• Be sure the materials have a logical" flow of ideas, 
a consistent format, an overview and/or # synthesis, 
grammatically correct language, and an interesting 
writing style . 
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Ho sure the publication has an appealing title, an 
attractive layout, a reasonable length, and a pro- 
fessional appearance* 



Caveats 



Quality may be sacrificed when the state-of-the art is 
underdeveloped, and the need is great- Real trade-offs 
can occur in development of high-quality products when 
resources are limited. The developer must balance the use 
of resources during the development with later dissemin- 
ation/implementation costs. 

Cuality can only be judged in relation to what is pos- 
sible at the time. Results of the needs assessment 
should be used to prioritize what is most important to 
ensure that minimum standards of quality are met 
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User Orientation 



Representatives of relevant audiences should be identified 
and involved in designing, testing, and using innovations. 
Primary audiences should' receive priority in dissemination. 
The resulting product should contain practical information 
organized in an easy-to-use format. 



Overview 

This criterion emphasizes the importance of conducting R&D „ 
with, by, and for intended users. Oser j orientation should 
consider both group and individual dimeits3>pns . Products that 
have been identified as being "relevant" and "easy to use" 
have been readily adopted in schools. Hood and Blackwell (1976) 
found an emphasis on interactive information 'gathering from local 
sources to be 'very important. Teachers and principals were more 
likely to use a product when they found it V likely to have the 
information they want" (Sieber 1981). To.be effective for 
primary users, R&D products should contain practical information 
organized into a useful format. Conceptual materials require 
additional .time for teachers to use in classroom activities. In 
contrast, instructional materials designed for direct, inde- 
pendent use by students are easy for teachers to implement. 
Conceptual materials may be more appropriate for researchersv 
administrators, and policymakers. 



Illustrative Case - 

An example of how materials evolve from teaching guides to 
individualized student modules was the Vocational Education 
Special Education Project (VESEP) in Michigan (Central Michigan 
University, 1974 and 1978). Almost one million dollars wes spent 
in three phases of product development. Ten instructional guides 
evolved from the first phase, supplemental materials were 
designed during phase two, and finally, in phase three, 
competency-based modules were developed for use by students in 
the classroom. When administrators, teachers, and students were 
interviewed during an impact evaluation of the Vocational 
Education Special Education Project (Bragg and Hull 1981), 
positive impact was usually found to have resulted from the use 
of the student modules. A comparison of the format of these 
materials indicated the more student-oriented the product became, 
the more useful they were for teachers. The materials from the 
first phase were conceptual in nature, requiring the teacher to" 
•further develop instruction for students • The materials from the 
last phase, however, were ready to use. The teacher had only to 
assign them to students in an appropriate manner and grade the 
competency that each student" achieved* 
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The ttMchors' oxp^ri^nce in usino the VKSHP materials 
emphasizes the need to know constraints of the primary user 
audience. Teachers do" not havo the time to develop materials 
when assioned to a- full classroom schedule* All the VESEP 
rater ials were attractively packaged and completed in a 
professional manner. However, an instructional format was 
rtissinq from the initial materials', thus these concepts had to 
be further developed by the. teachers . On the other hand, student 
modules were easy to use,, Project developers, should keep this in 
rind as they develop materials for classroom use. 

Qui del ines 

+ Identify one primary target audience. 

• Identify other relevant audiences, 

• Involve representatives of primary and secondary user 
audiences in desianing, refining, and testing the^ 
innovat ion . 

• Pes ign innovations with the user's ' needs , values , and 
abilities i nfen ind . 

• Involve usersN?n dissemination through networks and 
demonstration sites and by having users train other 
Msers. ^ 

• Establish guidelines for user involvement e«g«/ number 
of reviews, to maintain a balance between usefulness and 
scholarship . 

• Format the product for ea&y use through mechanisms such 
as illustrations, lists, advance organizers, 'summaries, 
indexes, modularized design, transparency masters, and so 
on • 



Caveats 

• User orientation may mean sacrificing quality and higher 
order concepts in an actempt to appeal to the widest 
possible aud iences • If some users have dif f icu lty , 
understanding the concepts presented, the content may be 
overly simplified in the interests of making 'the product 
oriented to the broadest range of users, 

• User orientation may detract from specificity because of 
attempts to address the needs of divergent users in one 

* product. " 
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Pevelopers should design a product to meet the needs of 
primary group of users; opportunities for broader appli- 
cation ' should be considered a secondary,* not primary, 
objective . 
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Strategic Dissemination 



Cost-effective strategies for disseminating R&D products 
should be based on characteristics of potential users, 
site-specific factors, and features of the product itself. 
Dissemination* should be strategic in reaching opinion 
leaders and influential organizations in the external 
environment . 



Overview 

Dissemination of a product should be strategic in relation 
to three dimensions: individual differences, contextual factors, 
and the power of marketing. First, potential users have various 
interests, styles, -and abilities. Some individuals are more 
oriented to live action and oral communication than to printed 
materials. Some users will be more predisposed toward a product 
than others. Strategic dissemination considers individual 
differences and fosters acceptance, ownership, and advocacy of 
innovations by specific individuals or groups. Second, con- 
textual factors are important. Schools may have more funds to 
purchase products at particular times. Interest in a topic may 
be boosted by a ^ public event. Dissemination should be timed to 
strategically capitalize on contextual opportunities. Third, 
different emphases in marketing the product will produce 
different results. In the commercial world, changing one word in 
an ad can boost sales by 40 percent. What pus communicated about 
a product strongly influences, who purchases it and how many are 
sold. 



Illustrative Case • 

A continuing goal of the National Center has been to assist 
disadvantaged youth to obtain the skills necessary for employ- 
ability. One National Center product related to this goal is 
Bridges to Employment (Winkfield et al . 1980). Many dis- 
advantaged youth are ^school dropouts and unemployed. Working 
through public schools would not reach much of this population ♦ 
Many programs for disadvantaged youth tend to be outside the 
public schools. In addition, program staff tend to be reluctant 
to use formal school-like documents with out-of-school youth. 

A strategic dissemination plan was devised to address the 
unique needs of out-of-school youth. A free, two-day workshop 
was conducted to help Neighborhood Youth Corps staff in 
Philadelphia to implement Bridges to Employment . All twenty- 
three workshop participants received a free copy of the printed 
product . 
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A follow-up study on the usa of this product was interesting 
(Bragg 1981). Only three of the workshop participants (13 per- 
cent) actually used the product after the workshop ♦ However, 30 
percent of the participants had implemented ideas from the 
workshop through presentations and handouts. In other word§, the 
product had been used only sparing Ly. But probably more impor- 
tantly, the information from the product had been communica ted i 
better orally than in writing* This case illustrates how a 
target population can be reached through verbal communication 
rather than printed information ♦ 



Guidelines 



Time the introduction of innovations to fit available 
resources, and compliment ongoing activities. 



Col lect market information on charac ter is tics of 
potential users and competing innovations . 

Design different dissemination strategies for different 
types of users in different types of settings. 

Devise different dissemination strategies for different 
types of products and information. 



Caveats 

Strategic dissemination requires precise information about 
characteristics of the product, potential users, competing 
products, and related activities. This complex array of infor- 
mation must be juggled when strategic dissemination plans for 
different users and situations are formed. Collecting such 
information can be a v/aste of time if not carefully planned, 
executed, and used. 
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Multiple Channels 



More then one channel for conveying information about 
innovations should be used. Communication should include 
mass media (e.g., direct mailing of brQchures) and inter- 
personal* channels (e.g., technical assistance). Normally 
w information duplication and overlap are an asset rather 
than a hindrance during the dissemination phase. 



Research on the change process has shown that repeated 
exposure to a new idea stimul/tes adoption. Individuals are more 
likely to purchase a product (if they hear about it from several 
sources. Once 1 a product is purchased, multiple channels of. 
communication can accelerate rts adoption by reinforcing use. 
Roger's (1962) model for the diffusion of innovations was based 
on five stages: awareness, interest, evaluation, , trial , and 
adoption. Each stage requires a different type of information. 
Subsequent diffusion models have incorporated two-way commun- 
ication. The Dissemination Analysis Group (1976) described four 
stages in their dissemination model: spread, exchange, choice, 
and implementation. The "exchange 1 ' stage incorporates feedback 
from users / which is an important consideration in the change 
process. Havelock and Lindquist (1980) build two-way commun- 
ication into every stage of dissemination. 



Illustrative Case 

The National Center has studied sex equity since 1966. 
Since that time, information on sex equity has been shared in 
numerous forms (e.g., research findings, products, # conferences, 
and technical assistance). Information on sex equity has reached 
the vocational education community through the following 
channels : 

• A national conference on the educational implications of 
women's work patterns 

• Training conferences for sex equity coordinators 

• Research on career patterns of adult women 

• Research on career plans of senior high school females 

• Survey of women administrators # 

• Development of a directory of women administrators 



Overview 
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• Development of a curriculum package on career options 

• Development of a parent handbook on career implications 
of sex stereotyping 

Development of annotated bibliographies and knowledge 
synthesis papers on equity issues 

• Development of a training package for sex equity 
coordinators 

• Development of a catalog of strategies for recruiting men 
and women into nontraditional careers 

m Development of sex equity training manuals for teachers 
in various occupational areas 

Multiple channels of communication have been used to 
influence the work of sex equity coordinators located in each 
state. A survey of sex equity ^coordinators (Brickell and Paul 
1979) revealed the following effects of multiple communication 
channel dissemination: 

• All recipients had read the publications 

• Two-thirds had attended conferences^ 

• Two-thirds had consulted with National Center staff 

• One-third had visited the National Center 

The typical coordinator was familiar with ten different 
publications. On the average, coordinators had participated in 
seven separate service activities. 

In thas illustrative case, multiple communication channels 
were quite/ useful in reinforcing use of sex equity resources by 
state coordinators . The many repeated contacts resulted in 
incremental improvements in sex equity for vocational education. 



Guidelines 

• Use mass media dissemination strategies such as direct 
mail of brochures., conference displays and announcements/ 
articles to create av/areness of and interest in a 
product. 

# Use interpersonal dissemination strategies such as 
workshops, technical assistance, demonstration sites, 
linkers, and networks to encourage trial use and 
adoption of a product. 

3<T 
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• <Y^ati» opportunit io« for frequent feedback from potential 
and actual users to facilitate ownership and continued 
use. 



Caveats . ! . 

Use of multiple communication channels requires access to 
primary audiences, availability of resources, and follow-up 
communications. Durinq this process, resources can be spread too 
thin across multiple channels. In addition, communication 
channels that taice lonaer to pay off may not receive sufficient 
attention . 
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Widespread Dissemination 



Innovations should reach as many potential users as possi- 
ble. Dissemination to individuals , in different roles, 
to diverse settings, and to many geographic areas should be 
emphasized. Secondary dissemination through workshops, 
reprints, libraries, ERIC, and so forth should be encour- 
aged. 



Overview 

There is a tendency to limit dissemination to geographic 
areas supporting development of the innovation. To minimize 
"recreating the wheel," products and ideas should be disseminated 
widely. The more potential users who hear about ayproduct, the 
greater the likelihood that use will occur. Not duly must a 
primary user audience be "saturated" with a product to stimulate 
use, but individuals in roles and organizations related to 
primary use should also be considered potential users. Wide- 
spread distribution stimulates diverse uses, as there is more 
variety in the roles and organizational settings of users. 
Widespread dissemination increases opportunities for use. 



Illustrative ca se 1 

Some of the most interesting and worthwhile uses of products 
have come from honrces unanticipated by product developers. This 
was true of the Emp loyabil ity Skill s series developed by the 
Center foij Studies in Vocational Education (197?) at Florida 
State University. This series of # six volumes focused on how to 
keep a job. It was implemented by using a series of workshops in 
secondary schools throughout Florida in 1977 and 1978. The 
following findings were somewhat surprising zo J nvestigator s 
studying the product's impact (Hull i9.81b) : 

• Students in Adult Learning Centers were using the series 
to prepare for .General Education Diploma (GED) exams. 

• The Telephone Industries Program at Seminole Community 
College was providing the booklets free to students 
enrol led . The 120 s tuden ts served per year each 
received ninety hours of instruction from the books. 
Interestingly, the College did not reproduce the 

book on Choosing an Oc cupat ion because they believed 
these students had ^already selected the occupation in 
which they wished to be trained. 
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# Every student completing the associate's degree program 
at Daytona Beach Community College was required to take 

a course on successful employment techniques using this • 
product. This included twenty to thirty students in each 
of five to seven classes each semester* 

• The, residents of Orange House, a facility of the 
Florida correctional system, wer^ us^ing the series to 
learn values. The residents were boys between the ages 

nof fifteen and eighteen who were one step away from a 
[(max imum secur i ty f ac il i ty . 



In Florida, the range of adaptive product uses was broad. 
The astute product developers defined the user group in order to 
include relevant technical skills for persons in different roles 
and organizations. The dissemination plan specified sending 
sample copies to these diverse audiences. 



Illustrative Case 2 

One R&D project at the National Center focused on iden- 
tifying new and emerging* occupations (Orth and Russell 1980). 
Results from this project were widely disseminated, especially to 
the general public. Distribution through the press and media 
created a "snowball effect" in terms of generating interest among 
the general population. Initially, the Ohio State University 
faculty newspaper, On Campus , described findings from the 
project. Then the city newspaper picked up the article and ran 
the story on the front page. Following this, a local radio 
station announcer interviewed the project's researchers 
concerning the results. This, coverage alone resulted in making 
about 125,000 individuals aware of new and emerging occupations 
identified through the research. However, this probabl^ 
represented only a small segment of the population who learned of 
the project's findings, as the results were communicated across 
the country through broadcasts from radi,o stations/ including 
stations in Atlanta and Los Angeles. In addition, the findings 
were published in national publications as well as in other 
professional journals, including Occupational Outlook Quarterly 
(Russell 1982). 



Results of the new and emerging occupations project were 
distributed widely due to the amount of interest shown by the 
general public. Of course, individuals affiliated with 
vocational education were also interested in the information. 
Administrators of state and local vocational . education programs, 
employers representing business and industry, and career 
education counselors found the information most useful. In all 
cases ,„ dissemination of the results increased individuals 1 
awareness of new and emerging occupations. 
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Guidelines 

• Give the primary audience priority in dissemination. 
* Reach as many members of this audience ~hs possible . 

• Disseminate the product to diverse roles within organi- 
zations . 

• Disseminate the prbduct to various types of organiza- 
tional settings . 

• Disseminate the product to mfiny geographic areas. 



Caveats 

• Be careful to explain any special requirements when using 
the product in order for it to be optimally effective 
under a variety of different conditions. 

• Widespread dissemination should not be sought at the 
expense of oih^r impact criteria (e . g .,. strategic 
dissemination). 
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Selective Implementation 



The introduction of innovations should be sequenced to meet 
the needs and unique characteristics of an adopting site. 
A process of mutual adaptation between .the site and the 
innovation should be encoOraged. 
' * I 



Ove rview 

Louis et al.< (1981), in the Abt Study of R&D-based 
products, found that the fit between a product and a local site 
was even more important than product, quality. If schools 
carefully define their local needs and find a product that fits 
those needs, implementation can occur without adapting the 
product . 

Whether or not a product fits a particular site depends on 
the following factors: « (1) site needs, (2) context, (3) socio- 
political values, ~(4) r.esource base, (5) timing, and (6) compata- 
bility with ongoing activities. 



Illustrative Case One 

In 1978, the Milwaukee School System was invited to field 
test some curriculum materials of the 'Alliance for Career and % 
Vocational Education. The Alliance is a consojrtium of twelve 
school districts formed by the National Center in 1973 to develop 
career and vocational curriculum materials. "Resource editions" 
of these materials had been adapted for mentally handicapped 
learners. Milwaukee was interested ih expanding its services to 
special students and believed the materials from the National 
Center were flexible, comprehensive, and easy to use. The use 
and impact of these materials were evaluated (Anderson and Hull 
1981) . 

The climate for special education is especially good in 
Milwaukee. Wisconsin has strong special education legislation. 
Public Law 94-142 is built on Wisconsin's State Law 115. The 
^trong work ethic in Milwaukee fosters community support for 
helping special education students become productive in the world 
of work. Since heavy industry supplies much of Milwaukee's job 
market, an array of jobs within the range of special education 
students 1 abilities is available. In 1978, the Milwaukee School 
District was searching for products to provide vocational 
education for special students. There was an excellent fit 
between characteristics of the product and the needs of the 
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Milwaukee School System. This closeness of- fit resulted in 
several benefits. First/ the ready-to-use Alliance materials 
increased systemwide adoption of a career education program. 
Second, the Alliance materials provided something tangible to us^ 
when bargaining with the. state for career education funds. It 
served as an enticement for establishing some new staff ~- 
positions. Third , the Alliance materials fit into Milwaukee's, 
long-range plans for establishing a new vocational center to 
serve only special education students, a plan ttiat is now 
underway. 



Illustrative Case 2 

The Rur.a l A merica Guidance Series (National Center 1§77) • 
provides another case .example of selective implementation. This 
series of sixteen handbooks was designed to help rural and small 
schools implement a career guidance and counseling program. 
Since development by the National Center was completed/ these 
materials have been adopted by over twelve hundred school 
district. * ^ 

A rural school district in Cashmere , Washington was one of 
the first users of the series. A case study of the innovation's 
impact was conducted by Modisette and Bormett (1981). Cashmere, 
is a small community with only a thousand students , yet it has 
established itself as a nationally recognised leader in career 
education. When the Cashmer^School District decided to adopt 
the* Rural America series, it had many innovative career guidance 
programs already in place. Castynere. was just beginning its 
Advisor/Advisee program, which established each teacher as an 
"advisor 0 to a group of approximately twenty students for the 
duration of their stay in school. Through this program, the 
usefulness of the Rural America series was recognized. 

The Rural America series is a process-oriented product that 
helps a school move through six phases of a planning-implemen- 
tation cycle. Consequently, it was an excellent companion to a 
programmatic innovation such as the Advisor/ Advisee program. The 
Rural America series helped the Cashmere schools District define 
the goal's of its new Advisor/ Advisee program. As one teacher 
said, "We had goals before, but after R ural America they were 
better defined and better formated to meet the needs of kids." 
The Rural America materials provided a framework for organizing 
and a process for implementing the Advisor/ Advisee program. One 
principal said, "Without the Rural America series we would have 
gotten to where we are, eventually, but by using the materials -we 
saved time and made fewer mistakes." 

Cashmere found that use of professional materials from a 
national research center helped sell the Advisor/Adviser program 
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to community members* As another principal said, "Some parental 
negativism toward the Advisor/ Advisee program died down when they 
found out 'we were using 'professional materials 1 to planing the 
program ." 



Guidelines 

• Encourage local sites to definetheir context, needs, -and 
resources systematically . 

• Time the introduction of products to maximize favorable 
conditions . 

• Utilize linkages to facilitate choices among R&D 
products ♦ 

• Provide assistance in fitting a product into existing 
frameworks, programs, and reward structures* 

• Begin implementation at one site by using 

a few. innovations rather than encouraging immediate 
systemwide acceptance of many innovations* 



Caveats 

• Developers should be prepared for the likelihood that 
users will make slight modifications in the product* 



• Change agents should monitor contextual factors that can 
influence implementation of the innovation to such an 
extent that the innovation loses much of its value • . 



Support Systems 



/ 




Systems necessary for full use of an innovation should be 
operational at the timfe of implementation. These systems 
are of three types: (1) personal resources (e.g., admin- 
istrative endorsement, personnel), (2) information resources 
(e.g., support material training, -procedures) , (3) physical 
resources (e.g.., facilities, supplies, equipment), and 
(4) financial resources. 



& 
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Supports-systems may include a complex network of people and 
procedures or may focus intensively on only one form of support. 
In ,,either case, support systems link product users with resources 
and Information to encourage effective product use. Support 
systems have an impact on product implementation. Without time, 
materials, space., and equipment, teachers frequently report 
problems when implementing products (Berman and Pauly 1975; 
Gross, Giaquinta, and Berstein 1971). 

Support personnel are the most important resource for 
implementation of innovations. Berman and McLaughlin (1978) 
explained, "To increase the likelihood that change will occur as 
a result of school improvement efforts, you have to involve a 
person whose function in part is guiding them through the 
experience." These "change agents" or linkers typically are 
teachers and intermediate-level administrators. They have the 
most contact with teachers implementing innovations. Intensive 
in-service training has consistently been identified as an 
important strategy for implementation (Fullan and Promfret 1977). 
This strategy provides teachers with models and experiences that 
assist them in ' implementing innovations (Berman an^ Pauly 1975; 
Crowther 1972; McDonald and Walker 1974). 

* s» 

C 

Rosenblum, and Molitor (1981) in a national study of 
research and development utilization ,„ found in-kind contributions 
from the adopting site (e.g., release time) to promote imple- 
mentation, kesults from this study suggest that staff release 
time and training by an external agent can significantly a I: foot 
the degree to which schools implement innovations. 



Illustrative Case 1 

This case study illustrates the personnel dimension of 
support systems in an impact study of several special education 
innovations, e.g. the National Center's It Isn't Easy Being , 
S pecial series and Michigan's Vocational Educatio n S pecial 
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Kdae at ion Proj-eot (Bragg, Hull , and Adams 1981). 'At one 
vocational technical institute in Minnesota, support service 
teams provided a broad ranae of services for teachers and 
handicapped students in vocational programs. Some responsi- 
bilities- were directly related to the implementation of R&D 
products from the National Center. Support services included 
inservice workshops, technical assistance, and written 
newsletters/bulletins. Handicapped students were introduced to 
R&D prolucts prior to use of the products in mainstreamed 
vocational classes. Teachers received individual help with 
product implementation. As a result, levels of use included the 
following : 

• Five R&D products introduced through inservice workshops 
were implemented by sixteen vocational teachers. 

• One product on individualized education programs was 
used by at least fifty-five teachers and four 
guidance counselors in sixteen .local high schools. The 
format for individualized education plans evolved from 
this product . 

• Almost 500 students were involved in individual tutoring 
sessions where the R&D products were used. 



Illustrative Case 2 ' ( 

This case study illustrates another critical support* system, 
accessibility of necessary materials and equipment. A product 
must not only be physically transported to the site of use, but 
it must also 'be accessible to the users at that site. Consider 
the case of microfiche readers that are used with ^Vital Infor- 
mation for Education and TT ork System (VIEW) ( Flor icta Division of 
Vocational Education 1970). VIEW provides current, accurate 
information about occupations in Florida. 

Beginning in 1970, ninety-eight mic of iche frame cards . 
containing 600 current occupations in Florida were made available 
to approximately 300 middle schools , *higTf schools, and other 
educational institutions. Each microfiche gave requirements of 
an occupation and listed institutions where training was offered. 
This system was coordinated with a computer-assisted placement 
service. It was possible for a student to go from the VIEW cards 
to Florida Employment Agency cards. 

An evaluation of the VIEW system (Hull 1931b) revealed that 
successful operation was dependent upon the availability of a 
microfiche reader. In most schools the microfiche readers were 
located in open, well-lighted areas, resulting in relatively high 
use of the machines. However, evaluators occasionally found 
machines in oat-of-the-way locations. Students h^d difficulty 
finding anl using the readers. Tn a few schools, funds for 
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reader-printers and supplies were not available. One school had 
only one reader, resulting in lines of students* The extent to 
which VIEW was used was directly related to the availability and 
placement of the microfiche readers and supplies. 

These findinqs illustrate the need to support operational 
products. Administrators must make sure users have access to 
products and necessary resources if innovations are to succeed on 
site . 



Guidelines ^ 

m Gain the endorsement of administrators for providing 
necessary support . 

• Provide training/assistance in using the innovation. 

• Support innovations through some local in-"kind contri- 
butions. 

• Ensure access to supplies, equipment, space, and 
materials . 

• Provide release time for staff to enable them to becrome 
involved in implementing the innovation. 



Caveats 

• Support services frequently require reallocation of re- 
sources, particularly of staff, time, and money. 

• Some schools may find it difficult to release teachers 
and/or administrators from present responsibilities to 
enable them to assist with R&D product implementation. 

/ 

• Many schools may find it difficult to hire -new per- 
sonnel to facilitate implementation . 



Cost Feasibility 



Information describing the innovation 1 s resource 
requirements should allow quick and easy estimates of 
costs likely to be incurred by an institution adopting 
the innovation. 



Overv iew 

Developers of innovations should be concerned about users 1 
need to estimate costs prior to implementation on site . y. 
Prospective users want to know (1) what supplies, equipment , and 
space are necessary; (2) what expertise is needed to operate the * 
innovation successfully, and (3) what the space requirements* are . 
These estimates should be made prior to implementing the inno- 
vation. Information necessary to make these estimates should be 
readily available. Some R&D reports include a section on 
implementation requirements. These requirements should be listed 
for different levels of implementation because an innovation may 
be implemented in stages. Full implementation of an entire 
innovation often occurs some time after initial use. 

Costs of adopting an innovation, per unit, may be reduced by 
sharing development expenses or by increasing the volume of 
products used. Other factors, such as ease of use, the amount of 
disruption caused by the innovation, and the availability of 
resources on a particular site, also affect the feasibility of 
implementing the innovation. 



Illustrative Case 1 

The Alliance for Career and Vocational Education was formed 
by the National Ceo her with a consortium of school districts in 
1973-74. The "districts pooled their resources, expertise, and 
money to develop and test career exploration, awareness, and 
planning materials for use at the elementary, junior high, and 
high school levels. The pooling of resources was a cost- 
effective way to supply each participating school district with 
'quality R&D products. 

The Milwaukee Public School System, a participant in the 
Alliance since 1974, has accrued cost savings from the Alliance. 
An evaluation of the Milwaukee program (Anderson and Hull 1981) 
revealed that the Alliance had proved to be a useful vehicle in 
expanding Milwaukee's career education program systemwide. 
Milwaukee's coordinator of the career education said "The 
Alliance was a vehicle for creating positions for carrying out 
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the Alliance programs. My position began that way." She went on 
to say that "The materials were something to sell/' In essence, 
the Alliance program not only made an innovation cost-feasible 
for Milwaukee but actually helped the school save money. 



Illustrative Case 2 

One product .that focused on improving education for special 
needs populations has been disseminated by the National Center 
since 1979. About four thousand copies of the product entitled 
Individua lize d Education Programs ( T RPs): 'A Handbook for 
Vocational "E duc ators (Phelps and Ratchelor 197°) tvvvo boon 
distributed throughout the country. Following this cxtensivo 
distribution, a study was conducted to* determine the impact of 
the R&D product on vocational education programs (Bragg, Hull and 
Adams 1981). Impact of the product was examined vi.a two on-site 
case stud les and via telephone interviews with^ about thirty users 
of the handbook. 

The product had a dual purpose that required two different 
levels of use. The first goal was to present basic information 
on the purpose for developing IEPs for handicapped students. 
When handbook users were asked about how they used this first 
section of the handbook, almost all explained that the infor- 
mation had been used to increase their knowledge about IEPs. 

The second goal of the handbook was to provide vocational 
educators with a format for writing IEPs, as well as with 
suggestions for appropriate administrative goals and respon- 
sibilities. Handbook users gave many reasons for not using this 
information. Certainly the lackv of "sufficient resources was one 
of the most important. In many cases, additional^ inservice 
training would have been needed to incorporate this information 
into the schools' procedures for writing IEPs. Most respondents 
indicated that their schools did not have sufficient funds to 
support this task. Thus, even though most respondents believed 
the information had increased their knowledge about IEPs, the 
handbook was not used to change practice because of- the costs 
required (funds, time, and staff) to adopt the information in the 
handbook. 



Gu idelines 

• Compute the cost of the innovation per unit of use and 
compare it with cost estimates of current similar 
activities . 

• Investigate the possibility of in-kind contributions or 
assistance from funding sources to reduce implementation 
costs . 
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• Determine if there are any implementation requirements, 
such as release time for teachers that should be 
negotiated with teacher organizations. 

• Look for cyclical considerations in the field (e.g., 
season of the year) that could influence the cost of 
innovation installation or availability of funds to 
purchase innovations . 

• Develop a schedule for lead time considerations when 
implementing the innovation. 

• Provide a range of costs rather than exact estimates. 
Inflation and other factors impinge on cost estimates 
made by developers. 



Caveats 

• Feasibility estimates should guard against inferring 
cost-related outcomes from the innovation. It is dif 
ficult to place monetary values on outcomes; what 
happens following implementation depends upon the 
resources at a particular site. 
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Multiple Patterns of Use 



Use of innovations wil 1 v^ry depending on the type of use , 
its intensity , 1 evel frequency , and extent . The users 1 
settings, roles, and demographic characteristics create the 
conditions for different types of use. Use of R&D by users 
other than the primary user audience should be encour- 
ag ed . 



Overview 

Teachers and other R&D product users should try different 
ways of using an innovation in diverse settings. Developers 
cannot anticipate all of the conditions faced by a user on site; 
therefore, imaginative use can contribute to the benefits of the 
innovation. 

0 

- Hall et al . (1975) have developed a "levels of use! 1 ' 
framework for analyzing innovation adoption. The framework 
defines eight discrete levels of use: (1) non-use, (2) orien- 
tation, (3) preparation, (4) mechanical' use, (5) routine use, 
(6) refinement, (7) integration, ari(3 (8) renewal. This framework 
relates to large-scale innovations in which significant levels of 
adoption are expected. It may be overly complex for analyzing 
smaller innovations, such as a single research report, instruc- 
tional guide, or conference. A more appropriate framework for 
analyzing a single product* s pattern of. use may be: (1) scan 
(2) read, (3) study, (4) reference, (5) share with others, 
(6) apply, and (7) adapt (Adams 1980). 

Often, there are concrete demographic and contextual 
factors that help determine patterns of use. Some examples of 
context descriptors of use are: (1) type of program, (2) grade 
level, (3) size of enrollment, (4) length of program, (5) program 
content, and (6) number of different products used. As should be 
apparent, 'patterns of use is a multi faceted concept that can be 
interpreted in numerous ways, depending on the nature of the 
product. 



Illustrative Case 

The strategic distribution of Performance- Based Teacher 
Education (PBTE), a curriculum developed at the National Center 
has resulted in multiple patterns of use. A case study of this 
innovation (Adams, Mac Kay, and Patton 1981) revealed that crea- 
tive* adaptations of PBTE occurred in many agencies across the 
nation. In colleges and universities,. PBTE was used for 
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preservice an<l in-service vocat ional\ teacher education programs. 
PBTE was used to train teachers in agriculture education, 
business and office education, distributive education, health 
education, home economics education, industrial arts, technical 
education, and trade and industrial education. At secondary 
schools, PBTE modules were used to upgradte the skills of 
practicing teachers through in-service programs. At 
postsecondary institutions, PBTE modules were used for staff 
development programs, most frequently as part^ of a comprehensive 
personnel evaluation and development system. In addition to 
education agencies, over 220 noneducation agencies (such as 
Caterpillar Tractor Company, IBM, and Union Carbide) used the 
PBTE nodules, most frequently as part of company training 
programs for improving instructional techniques. 



Guidelines 



Determine demographic and contextual factors that define 





patterns 


• 


Consider 




use . 


• 


Consider 


• 


Consider 


• 


Look for 



Caveats 

• The patterns of use must emerge from the data collected. 
The evaluator should be careful not to allow personal 
perceptions to influence the construction of pattern 
categories . 

• It may not be possible to determine the intensity of use 
without spending inordinate amounts of time observing or 
interviewing. Alternate measures of intensity of use, 
such as' lesson plans and curriculum outlines, may be used 
with proper disclaimers. 
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Time on Task 



R&D products should be used freqently enough and long enough 
for their use to become an integral part of current prac- 
tice. The amount of time that users actually use the pro- 
duct should be maximized. 



Overview N 

A number of evaluations during the past decade revealed that 
many innovations got poor marks because they were not implemented 
as planned. Research on school effectiveness ( Fisher , et al. 
1978; Stallings 1974) has concluded that the amount of time 
students 1 engaged in a particular subject is one of the most 
critical factors influencing student achievement. The amount of 
time individuals spend actually usinq an innovation is directly 
related to its degree of impact. According to Crandall, 
Bauchner, Loucks, and Schmidt (1982) the major factor producing 
change in classroom practice is the amount of classroom time 
spent on the new practice by the teacher. Classroom time also 
accounts for fidelity — how close new users come to reproducing 
the developer's ideal. Crandall also emphasized that the amount 
of time teachers spend on a new practice is affected by Lheir 
level of commitment. The more committed they are, the more 
classroom time they spend. Degner (1982) reported that specific 
assistance is another factor that increases the amount of time 
that teachers spend on a new practice. Specific, practical, 
teacher-focused assistance can help a teacher concentrate on 
using the new practice effectively. 



Illustrative Case 

The Career Planning Support System (CPSS) illustrates the 
importance of time on task. The Career Planning Support System 
(National Center 1978) was designed to help schools use a systems 
approach to develop a customized career guidance proqram. The 
innovation guides a school through six systematic steps: 

1 . 'Assessing needs 

2. Developing object ives 

3. Assessing resources 

4. Developing strategies and programmina 

5. Inplementina career development units 
G. Evaluating outcomes 



Handbooks and procedural nuide:, were designed to help schools 
conplete each step in developing customized career guidance 
programs for -their unique needs and resources* The innovation is 
quite flexible* Schools can spend very little *time or can spend 
extensive time on inpl ementat ion . Outcomes can range from a few 
supplemental career development units to a comprehensive career 
qu idance proaram . 

The significance of time on task to this innovation was 
especially apparent at the American Senior High School in HjLleah, 
Florida. This school began to use CPSS in September 1979. 
Purine* the 1979-80 school year, the school received a $5,000 
arant to provide teacher release time for implementation. During 
this year , the school made substantial progress in identifying 
nee^n and developina a few career development units. However, 
once tbt* nrant ended and teacher release time was no longer 
funded, the innovation came to a complete halt. As a group, 
teachers were in favor of the innovation and felt that guidance 
services were the weakest area in the school; however, without 
release time there was no incentive to work on it. The teachers 
agreed that if they could have taught four classes rather than 
five, most of them would have assumed a specific, ongoing 
responsibility for implementina the innovation. 

Cuidel ines 

m Avoid burnina up excessive time at the beginning of 
implementina an innovation (e.n., selecting an inno- 
vation, plannina for use, developing materials). Save 
time for activities after the innovation is in use (e.g, 
* implementation schedules, follow-up training, and 
procedural details). 

• M locate the greatest portion of time to the most 
critical parts of the innovation. 

• Provide release time for relevant audiences to implement 
the innovation . 

» Provide practice-specific assistance in using the 
innovation . 

• Maintain records of the amount of time spent on various 
parts and phases off the innovation. Obtain an accurate 
measurement of the time spent actually using the inno- 
vation compared to time spent plannina for use of the 
innovation . 
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Caveats 

• The mechanical use of an innovation can neqate the time- 
on-task variable. User enthusiasm can influence impact 
of. an innovation. 

• A commitment to use an innovation must be accompanied by 
a knowledge of how to use it and a belief in its ability 
to improve vocational educat ion . Other wise / the effects 
of time on task are likely to be diminished. 
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Integrated Utilization 



Use of innovations should be intensive and pervasive 
throughout the organization, drawing on personal 
commitment to institutionalize the product into organi- 
zational routines . 



Overview 

Organizational use of an innovation can be enhanced by 
professional commitment. Crandall et al . (1982) describes 
commitment as recognition of the importance of an innovation to 
one's daily professional life. Individual users' acceptance of 
an innovation contributes to organizational change. Crandall 
also identified teacher commitment as an important predictor of 
change in classroom practice. Sieber (1981) concured with the 
importance for individuals to internalize "knowledge and create a 
sense of "ownership . 11 He described this characteristic as a 
powerful incentive for change. 

A similar concept that is associated with integrated use is 
the notion of "institutionalization." This notion implies 
complete adoption of an innovation by an organization* When a 
product becomes institutionalized, people in the organization may 
no longer recognize the innovation as one that was developed 
externallyl Evaluators may even have difMculty finding traces 
of the externally developed product. 



Illustrative C ase 
- - i 

In the public schools in Carroll County, Maryland, special 
education teachers were using the Employability Skills for 
Special Needs Students (Carrol County Public Schools 1978) 
curriculum to teach students basic vocational competencies. Use 
of this curriculum was pervasive throughout the county. In f ac f , 
all special education teachers were using the curriculum in their 
classes. A study conducted by Hull and Bragg indicated that the 
curriculum was the foundation for the instruction of mentally 
handicapped students. 

There were several reasons for teachers' use of this 
curriculum. First, the curriculum had been developed within the 
county. Ten teachers were paid by the county to develop units in 
their schools. Four schools in the county had been involved in 
the development and field testing. This involvement created a 
sense of "ownership" in the curriculum among teachers. 

Second, once the curriculum was fully developed, all 
teachers in the county received in-service training. This was 
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particularly important for new teachers entering the system. The 
county coordinator for the curriculum development project 
assisted teachers with implementation of the curriculum. 
Implementation usually went smoothly, since the curriculum was 
designed for the county and was customized fo-r those schools ♦ 
* ' * 

Third , the curriculum was comprehensive . Development of the 
curriculum took two years, and teachers began to use it in its 
final form in *L978. Fhe curriculum had been in place in some of 
the schools for about five years at the time of the evaluation 
site visit. Teachers and administrators were ' familiar with its 
strengths and had adapted to its weaknesses. It had become a 
standardized course of study for handicapped students across the 
county schools. 

Fourth, the curriculum was complete; it was sequenced by 
grade level, thus promoting use with students. In addition, the 
curriculum-referenced resources were readily available in the 
county. In summary, the curriculum had been fully integrated 
into the school system. 



Guidelines 

• Encourage adoption of the innovation over a long enough 
time period to create user identification with it . 

+ Develop the innovation fully so it is complete and is not 
jdependent on resources or expertise outside of the 
organization. 

• Encourage users to adapt and refine the innovation 
continually to maximize product-site fit. 

• Encourage enough users in an organization to implement 
the innovation to create a "critical mass" of support. 

• Encourage users to reevaluate their use of the innovation 
and to review their use through exploring new develop-" 
ments in the field. 



Caveats 

• It may take a long time for an innovation to become fully 
integrated into an organization. Evaluators and others 

* should not expect too much, too soon. 

• Speed of integrated use is directly related to the size 
and complexity of the organization. Large, complex 
organizations usually assimilate innovations more slowly 
than do small, simple organizations. 
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User Satisfaction 



The R&D product and its implementation should meet users 1 
expectations and should result in a positive user attitude 
toward the product. User satisfaction may be indicated by 
product advocacy and/or creative adaptations . 



Overview 

Users' expectations can be met a number of ways. Rogers * 
(1962), Zaltman (1980), and Rothman p.974) have spoken of the 
compatibility of the innovation with the user's established norms 
and values. A, similar but more important idea is relative 
advantage, which is when an innovation exhibits merit a£>ove and 
beyond the existing situation. In other words, it is superior to 
the ideas it supersedes (Rogers 1962). As a result, the user 
tends to experience personal satisfaction from implementing the 
innovation . 

■ i 

Illustrative Case I 

,A two-volume handbook on conducting follow-up studies, 
entitled Guidelines and Practices for Follow-up of Former 
Vocational! Students (Franchak and Spier 1978), wa^s published by 
the National Center. This product was rated especially high on 
"user satisfaction" and provides a good case for illustrating 
this criteria. 

ThKoJgh a f ollow-up^evaluation (Brickell and Paul 1980), a 
group of practicing avaluators who had read the two handbook 
volumes were interviewed. One question asked the respondent^ to 
spend $1,000 hypothetically on ten different kinds of evaluation 
data, which included test scores, , review teams, observations, 
manaaement information systems, and follow-up data. Follow-up 
data came out as the big winner, receiving 60 percent of the 
funds. According to the spending pattern, follow-uj^ data were 
viewed as the most important evaluation data. Users saw a 
critical need for information on' this topic. 

Users also reported being satisfied with the coverage and 
quality of the handbook. They cited the volumes' thoroughness, 
readability, organization, and comprehensiveness. A particular 
strength of the handbook was its assimilation of many sources of 
Information about follow-up studies in one book. Virtually no 
users could think of any other publication that they felt was 
better than this handbook on the topic of follow-up studies. 
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Guide-lines 



Caveats 
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Determine if the product effectively met needs 
norms, and values of the intended users* 0 

Assess the product's relative advantage over other re- 
lated products in use, 

Determine if users recommended the product to their- 
friends . 

> 

Solicit users 1 impressions of the major strengths and 
weaknesses of the product. 



Users may be satisfied with using existing, low-quality 
products rather than adopting a higher quality innovation 
that is complex and disruptive. 

Some users may require more time to become satisfied with 
a product than others (i.e., early vs. late adopters) 
than others. 



6 



60 



Individual Growth 



Innovations should contribute to changes in individuals' 
attitudes, knowledge, and/or performance. 



Overview 

The use of innovations should, contribute to individual 
growth. Effects may filter through organizational layers of or 
nay be interpreted by the perceptions of textbook writers, but 
R&D resul'ts should ultimately change individual lives. * These 
changes may be cognitive ( Bloom' 1956) , affective (Krathwohl, 
Bloom and Masia 1964), or action-oriented (Harrow 1972). 
Developers and distributors should specify objectives for 
individual growth that are relevant to individual use. 

Some educational research takes years before it is trans- 
lated into useful programs. Reinforcement plays an important 
part in instructional material development today. Such funda- 
mental variables as reinforcement were first demonstrated in 
animal experiments before being developed and tested with humans. 
The contributions of B. F. Skinner's investigations to programmed 
instruction have been documented by Suppes (1978). 

The use of individuals as sample units and/or as units of 
measure gives increased - flexibility to impact studies. Indiv- 
idual scores on tests can be computed as gain scores in pretest, 
posttest measures or used as covariants to screen out unrelated 
influences in the impact design. Experimental" methods have been 
used to document the impact of R&D on persona^ growth. 



I llustrative Case 

One successful use of R&D findings to bring about individual 
qrowth is the National Academy's In-residence Program at the 
National Center. This open-entry, open-exit program provides an 
opportunity for vocational educators to use the resources of the 
National Center at cheir own pace and expense. In-residence 
programs range from one week to six months on topics selected by 
the participant. The primary factor in the participants' growth 
is access to knowledge resources. Library books, National Center 
publications, Educational Resources Information Center (ERIC) 
microfiche, and National Center staff interact to support 
participants' growth. Learned facts are buttressed by group 
interactions with other program participants or with visiting 
authorities on topics of special interest. 
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A five-year follow-up study (Hull and Hassan 1982) revealed 
several areas of program impact ,on participants . Three-fourths 
of the in-residence program participants reported significant 
growth in personal and professional goals* As a result of the 
in-residence program, they gave presentations, workshops, and 
courses ; hared information through personal consultation ; 
received career changes or promotions; and wrote one or more 
publicatons. Personal growth also usually resulted in changes 
made by, individuals in their organizations. These changes 
included policy recommendations, improved curriculum offerings, 
initiation of new research, improved instructional methods, cind 
changed organizational structures. This contact with research 
findings in a scholarly atmosphere enhanced the likelihood of 
personal growth. The pace of the participants ' learning was 
correlated to their needs and desires . 

In-Residence participants were asked to estimate the number 
of individuals they had personally influenced as a result of 
knowledge gained through their program. Eighty percent of the 
sixty-four participants were able to identify eighty-eight 
thousajid individuals (mostly students and teachers) who had been 
influenced by the in-residence program activities. These people 
represented a secondary effect of individual growth. They were 
people who had contact with the primary participants and who grew 
as a result of this contact. 



Guidelines 

• Specify intended outcomes for individual growth. 

• Assess attitude changes through qualitative questions 
(e.g., V/hat are your feelings about the innovation?) 

«> Assess changes in knowledge through written tests 
and structured interviews . 

• Assess changes in performance through performance 
tests and direct observations. 



L? /eats 

• Many factors impinge on individual learning. Rational 
information may not be the basis for decisions affecting 
individual growth. 

• Changes in attitudes are particu 1 »r-iy difficult to 
measure. Prolonged exposure to an innovation may be 
required to change attitudes and beliefs. 
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• Short-term gains in student learning may disappear when 
students are retested several months later. 

• Comparison groups for field experiments are particularly 
difficult to locate for impact studies. 
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Organizational Change 



R&D should contribute to benef icia 1 changes in organiza- 
tional policy/ programs/ practices, and/or structure, as 
reflected by savings in costs and time over current 
practice* 



Overview 
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Organizational change involves a wide range of effects that 
occur in educational institutions as a result of using an inno- 
vation. Organizational change may be hampered by many inherent 
characteristics (e.g., the size of the organization, the number 
of years it has been in existence, the length of time staff have 
been employed, and previous disposition toward change) (Downs 
1967). Educational organizations vary widely on these factors* 

Change in an organization implies more than routine adoption 
of a new product. The use of the innovation has gone beyond 
adoption to produce some type of improvements, according to Yin, 
Heald, and Vogel (1977). Cost-benefit measures may be employed 
to determine which program change is best for an organization 
(Rossi and Freeman 1982). 

The importance of multi-year funding for organizational 
changes cannot be overemphasized. Adequate resources (e.g., time 
and money) allow staff to be trained and sufficient expertise to 
be developed within the organization. An innovation may disrupt 
routine and disturb people's domains of influence. This is why 
sustained support, both interpersonal and financial, is indi- 
spensible for organizational growth and renewal. The case study 
that follows illustrates not only organizational change but; also 
inter-agency support for that change. 



Illustrative Case 

The case of the Career Resource Centers (CRCs) in 
Pennsylvania illustrates the need for sustained financial support 
(funds ) and involvement of staff throughout a school . Sixty- f out 
CRCs were established by the Pennsylvania Department of Public 
Instruction in 1972 to provide current, accurate career 
information. A set of guidelines tot these centers had been 
developed, based on experiences in five pilot sites* The 
following observations, taken from a study of product impact r 
(Hull 1980), emphasize procedures that enhanced the 
institutionalization of the innovations: 

. . / 

• Establishing the CRCs was an interdepartmental initi- 
ative. The Pennsylvania Research Coordinating Unit took 
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the lead with the tacit approval and cooperation of the 
state's Vocational Guidance Section and its Division of 
Career Education. 

• Funding was sustained and prorated, based on the number 
of years the particular CRC had been established. All of 
the costs for the first year of operation were paid by 
the state. Costs in subsequent years were gradually 
turned over to he local school districts until all or 
most of the costs were being paid by local districts at 
the time of the impact study. 

• CRCs were staffed by local agencie.s. Differentiated 

staffing patterns characterized the centers. Most had a 

guidance counselor or occupational specialist in charge 

of. the CRC, with teacher aides or students helping with 

"use of the equipment and/or shelving of publications. 

> 

• Systematic contact was made with teachers to solicit use 
of the- CRCs. This use took many forms (e.g.,. .career 
awareness, assignments in class, presentations by the 
CRC director, m and drop-in use of the CRCs by teachers). 

This innovation resulted in organizational change because many 
separate activities came together at the Career Resource Centers 
to provide a viable, integrated mechanism for the school. 



Guidelines 

• Specify intended outcomes for organizational change. 

• Compare costs of operation after the innovation has 
been implemented with prior cost records. 

• Look for structural changes, e.g., in the organiza- 
tional chart, for indications that the R&D product has 
had an effect on the organization. 

• Examine policy statements for information from R&D 
reports . 



Caveats 



Changing groups of people in organizations is more 
difficult than influencing a single person. 



• Structural changes take more time than other types of 
changes and may not be reflected in immediate benefits. 



• Each organization exists in a unique context that must 
be considered when introducing research-based inno- 
vations. 



# Organizational change tends to require an extended time 
period. 
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Societal Contributions 



k&D should contribute new and significant information with 
the potential to advance knowledge, improve current practice 
and/or influence social systems • 



Overview 

The impact of research On education has been summarized by 
Suppes (1978) in a series of case studies* One is struck by the 
somewhat fragmented nature of the impact of research as well as 
by the rather optimistic tone of the reports There are areas 
where research has had great impact on practice. For example,, 
the theory of mental tests and pupil classification has been 
applied in the public schools, Thorndike's theory of learning 
(1913) has been applied successfully to the teaching of 
arithmetic. In vocational education, the use of experienced- 
based cooperative education has influenced the* way employers 
acquire skilled labor. 

Systematically derived information from R&D is intended., to 
briny change and improvement to education, but it is difficult to 
sustain the research funding needed to resolve important, 
persistant problems. In a report to the National Institute of 
Education by the National Academy of Science (Kiesler and Turner 
1977) recommended more funding for fundamental research relevant 
to education. This was to be for long-term research intended to 
affect broad sections of society. In practice, major changes in 
society come from a combination of factors. Strategic funding of 
studies is required to maximize the benefits from scientific 
knowledge. 



Illustrative Case 

The Performance- Basea Teacher Lducation ( PBTE) curriculum is 
an example of an innovation that has facilitated major changes in 
teacher education. PBTE rode the cres't of a major reform 
movement in education* The initial development of PBTE was just 
ahead of the competency based movement. Consequently, the 
product was ready to meet a growing interest in and need for 
performance-based materials . 

Based on in-depth interviews with forty-five college and 
university administrators and faculty '(Adams, MacKay, and Patton 
1981), PBTE was found to precipitate significant changes in many 
vocational teacher education programs* in a period of declining 
aeed lor secondary teachers, the PBTE curricula helped university 
vocational education departments survive the crises of funding 
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cutbacks and faculty retrenchment . Because PBTE lends itself to 
an individualized approach, universities were able to provide 
field-based programs to larger service areas. PBTE also made it 
possible for universities to attract community support, receive 
state funds for PBTE program development, and maintain 
enrollment. PBT^C ha-s accomplished the following: 

• Increased access to vocational teacher certification by 
providing self-contained instruction especially that was 
useful in rural and isolated areas 



• Increased flexibility in getting immediate help to new 
teachers whenever they were hired ' 

• Increased productivity of teacher education programs at 
some institutions by shortening the time required to 
certify vocational teachers and lowering costs by using 
differentiated staffing 

• Reduced variability and increased accountability of 
vocational teacher education curricula through 
standardizing the skills vocational teachers we're 
required to master 

• Significantly changed the role of the university teacher 
from a classroom lecturer to a learning facilitator 
working with students on a one-to-one basis 



Guidelines 



Specify societal effects in impact studies. 

Pick topics carefully when assessing societal impact 
because timing is important. 

Look for preliminary indicators of potential widespread ' 
changes in an institution so these areas can be 
supported . ^ 



Caveats 



The impact of* R&Dr on society may be the result of some 
unpredictable factors (e.g.; an idea whose time has come, 
or that is beyond the control of most individual in- 
vestigators . 

Societal effects are pervasive, take longer to emerge, 
and are difficult to measure in any single impact study. 
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CHAPTER IV 
APPLICATION AND USE OF CRITERIA 

Persons most likely to use the R&P impact criteria discussed 
in this document are research administrator\ and others concerned 
with usina R&D products to improve vocational education programs. 
Project directors can use these criteria to monitor development 
and dissemination activities. Much of the R&D impact in the 
later staaes of proqram improvement is determined by activities 
in the development and dissemination stages. A linear continuum 
ties toaether what otherwise may appear as discrete activities. 
This is true for k&l> projects in local education agencies as well 
as those in state departments, universities, and research 
centers . 

'Attainment of these criteria improve the chances of an R&D 
.product to create significant change in vocational education. * 
This chapter discusses some ways a research administrator may 
operat ional ize R&D activities to meet these criteria. The 
criteria in each staqe will be discussed separately. 

R&D Program Improvement Stages 

research administrators who look for innovative products to 
share with others generally apply the criteria of systematic 
development, hiah quality, and user orientation to the products. 
Many research coordinating units have mini-grant programs that 
award money to individual teachers to develop ideas for possible 
distribution to other teachers. In examining the products from 
these nini-orant proarams, research administrators should look 
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, for either a product tested over a long period of time by 

teachers or for a product used by a number of teachers to satisfy 
the criterion of systematic development. Review cycles that 
incorporate recommended changes into the product after trial use 
improve its chances of havinq a positive impact. Development, of 
checklists to rate product quality on different dimensions aids 
in the revision process. 

One v/ay to ensure attention to user orientation is to hire 
practitioners either to help develop the product or to review it. 
Field tests of product quality with students in a local education 
aqency settino are particularly important for instructional 
materials. R&D project directors should indicate in their 
technical plans the relevant criteria for impact assessments. 
The careful deployment of field site testing materials, for 
example, can assure geographic distribution of knowledgeable 
persons in the field who can be called on later to offer 
technical assistance to sites adoptinq a product. 

Dissemination 

Sometimes project d i rectors are responsible for d isserr- 
inatino the results of research or development. When this 
happens it is easy to miss persons who need the material. The 
criteria of strategic dissemination, multiple channels, and 
widespread dissemination are more likely to be met if the product 
is disseminated by the sponsoring organization . Decisions to 
share a product with a particular client qroup can be made 
stratenica] 1 y to satisfy cost-benefit considerations. T hero 
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nro 'po^'pr epouah oop i to oo .^rourn^; nor is dissemination the 
Inst ?t w o* p development activity. Father, it should be 
f lower' ps on orpcrt unity to pj are proieot results in the hands of, 
the peorle who need them. 

Information about a rrodpet can he condensed and spread to a 
wider audience bv usi^o pass media than by relyino on word-of- 
~oi-t^ recommendations r ror the product. Mass media information 
,»llrw« interested person? who are not in the primary audience to 
rfmu^t a oop^. whereas t^e burden of see>ino P&r information is 
op the usor, a rro^rf disseminator can mabe jt easy to find 
relc\v>rt information. 

Imp t or opt at i op 

rissomj nat i no an P£P product with a potential for h iob 
impact docs not onarantee results. n>o products have to be used 
ape 1 , hefore thev cap be used, they must be implePted. Adoption 
rites ^i r fer ir their abi 1 i t v to support innovations. Fofre can 
p Ccer + op rf K r prodprt in its initial form. Others have to modify 
tho nro^vct for it to he acceptable. Feasibility studies of 
oosts li^elv to be incurred in operation of the innovation are 
t^rc^F^rv to insure smooth imp] ement at i op apd operation, 
^oohric*] expertise and administrative endorsement are other 
wrrort variables that can ma^e a difference in the innovation's 

/v pew idea ^mpo to ca i n a *"oothoJd in established practice 
je r r ^np ; it needo f^vor^le conditions. ^his is why selective 
i-i iprfPf.uion i<- rooommepdod. *<ot all parts of an organisation 
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accept an innovation equally well. The services of volunteers, 
for example, are recommended to provide a favorable context for 
early use of an innovation . Innovations with adequate support 
systems stand a better chance of creating a positive impact than 
those without such support . 

Uti 1 izat ion 

Innovations implemented in various degrees will result in 
multiple patterns of use. Some teachers will be better prepared 
to use a new idea or simply be more naturally inclined to use a 
particular product* For example, a curriculum plan using 
teachers as resource personnel to answer students' questions 
appeals to some teachers more than others. Teachers who are 
confident, interactive, and not tied to reference materials ar£ 
likely to use such an innovation. ^hus, style of teachinq can 
become a selection factor in using an innovation when the 
directions of use cannot always be anticipated. 

In onler for an innovation to have a positive impact, it 
must eventually be integrated into the total operation of the 
adoptino organization. Selective implementation may be 
necessary, but the innovation can not remain an isolated part of 
an educational system. l%hether a product is an innovative 
manaaement procedure or is a new way of teaching employabl 1 it y 
skills, it must be used for a specific lennth of time for impact 
to take place. Time on task is another important consideration 
if changes are expected in individuals, or qa nidations , or 
soci ety . 
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♦ f Oft e 

rpfpcts of the use of an innovation should he ir»onitored and 
rrrorrV^ r or two purposes: aooountahi li ty and plannino. The 

/ 

criteria of individual orovvth, user satisfaction, ornanizat ional 

r> <->nce, ard societal rop + rihut ions should not he applied to every 

t 

D ^ r rroSuc*. c or * e proc^urts, such as a technique for usino a 
p^rr^r correct J v in oa rpent rv" shop, relate only to skill 
*o\'o i oiropf of tbp individual . other products, such as 
. r r j f oruT-rp^ erpneed topt jpr, h<->vp potential for chanoina the way 
classes >->r<* or nap ifed. Fopp products, such as performance— hj?sed 
f^i^^r p.'iipm i<^, ovt'p have potential for mahino the educational 
process ^ore efficient and increasinc the c'ont r i hut i op of 
r"Voa**iop to society, (dances shoo I'd he recorded, because 
rpse^rc^ ac lw> i p i st rators need to Vpow if the monev and time used 
ard diffuse P&P rroducts have heen worthwhile, 
^hore rust v p a retur n to society for investments in P&P to 
poptirup. c uc> herchrarVs of account ahility serve a douhle 
rrrrop^. "ot orlv do t^ev indicate the effectiveness of current 
•\f r , t-^ev a ' so ' provi dp a baseline for measurino future P&P 

o^fprt^. Tppao** studies of Pf.P products should indicate if a 
nr*( i p~ hp*? > *>oi at h\->Qt- rartiallv resolved. Tn this wav, 
j "i act ^tp^i^s provide orfoino needs assessments. 

Assessing Impact 

Ac^occpent- of irpact typical iv reouires multi -method 
^rrr^^r^os. Trp^pt- ?V»opid hp assessed at each staoe of the P£ T 
. f > ^ir-iTt' « 1 ^rrvj'Vs sample approach ps t^at mav ^e vised to 
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Figure 4 

SAMPLE APPROACHES FOR ASSESSING IMPACT 



DEVELOPMENT 



DISSEMINATION 



IMPLEMENTATION & USE 



EFFECTS 



Critical reviu'% ' : 

Users and experts 
can be involved 
at various points 
in the development 
process to critique 
the innovation 
against performance 
standards • 



Recordkeeping & 
document analysis : 

Establishing a 
system of records 
is especially useful 
for tracking 
dissemination of 
innovations to 
various users, organ- 
izations, and loca- 
tions . 



Q uantitative s urveys : Qualitative Studies 



Surveys that employ 
structured questions 
are useful for identi- 
fying patterns of 
implementing and using 
innovations . Surveys 
can also help identify 
key users and innova- 
tions that are likely 
to produce effects. 



On-site case studies- 
using direct obser- 
vation and personal 
interviews are useful 
for discovering the 
effects of an innova- 
tion. Observing an 
innovation in action 
in its 'natural context 
provides a rich 
description of effects, 

Experimental studies : 

Once a clear descrip- 
tion of the effects 
of an innovation has 
been established, " 
experimental, studies 
may be useful to 
verify causal infer- 
ences. 
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assess impact throughout the PhV process. sSome inpact approaches 
(e.-i., those in fiqure 4) can be used at every staqe of the R&D 

A f ew qene ral pr inciples for des igninq impact assessments 
>re as fol ) cws : 

1. Specification of data requirements is necessary for 
imact assessment. As the requirements become nore 
specific, nuant itat ive methods (e.c, survey aues- 
t ionnair^s) become more useful. However, some 
ii">art can be unanticipated or difficult to detect. 
!V>r piu*h inquiries, qualitative methods (e.c., 
successively focused interviews) yield the best 
i r f or rat i or, . 

?. Resources (e.q., tire and money) for impact studies 
are usually in short supply. The best combinations 
of approaches for assess inq impact are those that 
rrrvi-^ the rinimur required information at the least 
(v, *;t . 

3. Ti^e ir> necessary for effects to occur. A complex 
innovation requirino extensive change in the adopt inq 
oraanizat ion will require more time for its effects 
to emerao than will one requirinq little organi- 

zat iona 1 chance . 

4. An alliance of both quantitative and qualitative, 
evidence should be used to assess impact. Data from 
bot-h approaches may be rerned in the same report, 
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providing that the integrity of each data type is 
maintained . 

5. Studies that facilitate cross-site synthesis should 
be more effective in determining impact than an array 
of individual studies would be. 

6. Impact studies should allow for unanticipated and 
' ^iosyncratic discoveries . 

The Appendices contain sample instruments for assessing the 
sixteen impact criteria presented in this report. Appendix A 
contains a User Survey that employs structured quantitative 
questions to examine various dimensions of impact. Appendix B 
contains an I nterview Guide that employs a series of open-ended, 
qualitative questions for describing impact. These instruments 
represent two examples of the kinds of impact assessment tools 
that could be constructed. Many of the items have been drawn 
from impact studies conducted at the National Center. * Each 
impact study demands a unique assessment tool . The relevance of 
the sample instruments to a particular impact study would be 
uetermined by the purpose of the study, the user population, and 
conditions on site. 
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Summary 

The state of the art for conducting R&D has advanced in the 
past twenty years. A knowledge base has evolved that, if 
•ccnsistantly mpl enented , could enhance the payback from the R&D 
dollar substantially. A technology is developing to insure* 
Planned change as a result of programmatic R&D. Society can no 
IrtKor afford wasteful and unnecessary R&D that fails to improve 
vr*Mti»>n.il education programs. Important performance criteria 
rust be developed and net to assure wise use of R&D resources. 

This report presents an organized research-based taxonomy of 
sixteen criteria for creating impact. The taxonomy is based on 
the prerxse that impact potential can be incorporated into pro- 
ducts at each staae of their development and use. The stages 
t;t'cd in this report are development, dissemination, imple- 
mentation, utilization, and effects. If a product meets the 
firrt twelve criteria in the four formative stages, it gains 
^oteytial c or creatine impact in the fifth stage, effects. 

Purina the development stage, the product should be 
(1) systematically developed , (2) high nullity , and (3) user 
oriented. The dissemination stP-ie must use (4) strategic 
dissemination and (5) multiple channels of co mmunication tc 
increase the likelihood of (6) widespread dissemination of 
products. The implementation staqe must be (7) selective and 
(e) cost feasible as well as assuring necessary (8) support 
systcrs are in place for product use. v The utilization stage 
^ho^ir*' support (10) integrated use and sufficient (11) time on 
task so that (I?) multiple patterns of use result. 
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If formative impact criteria are met, an R&D product has a 
much greater chance of producing summative impact when the 
product is disseminated and used. Four criteria are listed in 
the effects stage of the taxonomy. They are (13) user satis - 
faction / (14) individual growth / (15) organizational change / and 
(16) societal contributions . These sixteen criteria should be 
systematically and incrementally met throughout the R&D program 
improvement process to result in maximum impact. 

The sixteen R&D impact criteria presented in this report 
need to be refined and further developed. Next stages include 
use of the criteria as a collective set of impact indicators. 
R&D project directors could assess project activities to 
determine if the criteria have been met at each stage of the 
process. Sets of questions should be developed to interpret each 
criterion for particular products. The sets of questions in the 
appendices provide a point of departure for writing measurement 
items; hut, they do not substitute for impact assessment items 
written by a person knowledgeable about the particular R&D 
innovation. Trial use of these criteria are likely to lead to 
further refinement. Additional criteria may be needed or some 
may be deleted. Even the program improvement stages themselves 
may be modified. 

One of the most important concepts to emerge from this 
report is the notion of "impact potential." This concept may 
prove to be very helpful as researchers try to anticipate the 
amount and direction of change resulting from product use. 
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Inpael potential could become <i key indicator in determining the 
amount of money that should be spent to diffuse a R&D product. 
In any event, these criteria merit serious consideration by 
researchers and evaluators as they continue their quest to 
maximize impact from R&D products. 
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APPENDIX A 



USER SURVEY OF INNOVATION IMPACT 



S3 



USER SURVEY OF INNOVATION IMPACT 



1. ' v o what extent on vou think the innovation net the followinq 
criteria? (OhecV one for each item.) 



Criteria 


Mot 
at All 




To Some 
Fxtent 




To 
Great 


r?s*V on irportant pppi'p 




? 


3 


4 




I'uilt on exist ina knowle^ae 




2 


3 


4 


5 


Invo! vp' 1 users in r'esian 




2 


3 


4 


5 


Puilt on theory or conceptual 
f r.TPwor^ 




2 


3 


4 


5 


Systematically developed 


1 


2 


3 


4 


5 


Carefully tested 


1 


2 


3 


4 


5 



2. "pv vou l (1 vou ra*-e the nualitv of the innovation on the 
fol iowina criteria? (One ratine per criteria.) 

\'ot Very 

Criteria ftppl i caM e roor Fair Goor? Good Excellent 

p?lp\'3nce to rv 

neer>R 0 12 3 4 5 

0 1 2 3 4 5 

0 12 3 4 5 

0 1 2 3 4 c 

0 12 3 4 5 
r 

G\T *AT r OI'M ITY 0 1 2 3 4 ^ 
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3. <r o what extent were vou involved in the following 
act ivit ies 9 

F XT PINT OF INVOLVEMENT 



STAHE OF 


N T ot a t 












I^OVATIP^ 


M 1 




Fxtent 




Extent 


Appl icaMe 


f*es io n 


X 


p 

<- 


-\ 






0 


\ 








j 






revelOfVent 


1 


? 


3 


i 




0 


Test ino 


] 


7 


3 






0 


r»i sserr>i nat ion 


] 


< 








0 


Peceivi no tra ini no/ 














terbnica I ass istance 


] 


? 


3 


4 




0 


Giving trainging/ 














technical ass istance 














to other users 


1 


2 


3 


4 


5 


0 



4. How did vou learn about this innovation? (Check all that 
apply* ) 

lJ Peceived broch ures /f 1 yers 

I I Pear 1 product announcements/ 

articles in periodicals 

D ^aw djsplavs at conferences 

I — I Participate^ in workshop (s) 

D Peceived technical 
ass ist ance 



u Visiter? a demonstration site 

D Worked with a linker 

D Involved in a network 

n other (please specify) 
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6. InUcate the number of people for each role and organization that 
you have personally assisted in becoming familiar with this 
i nnovation . 



■ REALIZATIONS 



h, 



! »• t i>r»iat iotM 1 

" ational education 

St odi^at ion 

! . >oa 1 education 

Pr mary/sooondary 

r ^stSL-con-Iary 

Ccl 1 010/ ur.i varsity 

Pibl ic sector 

it ivate sector 



►search 



ROLES 

Administrator/ Teacher/ Student/ 
Manager Staff Client 



TuTAI 



f 



6. What were your costs for implementing the j innovation? 

j 

Purchasing materials $ ( 

Salaries for release time $ ' J_ 

Purchasing or renting 

supplies , equipment and space $ 

Other services/eg. computer time $ 

Other $ 



ERLC 



35 QO 



7. 



"ow important were the followina factors in influenoino 
ippl e^ent at ion of the innovation by your orqan i zat ion? 
(Circle vour ratine for each ^ac^or.) 



FACTOP PFOPFF OF TP PORTA NCF 

Tow Med i up Fioh 



leadership of tor 



a^r^i ni st rat or 


1 


? 


3 


4 


S 


T eacher ' s orient at ion 
to chanqe j 




2 


3 


4 




Availability of ^/nr?s 




2 


3 


4 




Availability of tit*e 
for ippl epent at ion 




2 


3 






Size of the oroanizat ion 




2 


3 


4 




teacher's influence in 
decision pakina 




2 


3 


4 




Other 




2 


3 


4 





b. 



To what extent were adeauate support systems available for 
ipplepentina the innovation? 



ADFOUACY 



mvpp op prppopT 



N T Ot 

at A3 1 



To Some 
Extent 



To a 
Great p xt ent 



A dni nistrative endors epent 


1 2 


3 


4 




Personnel j pvo) ved 


1 2 


3 


4 




Support raterial 


1 2 




4 




^rainina • 


1 2 


3 


4 




Funds 


i 2 








^auippent 


1 2 


3 


4 




Fuppl i es 


1 2 




4 




<^tber 


1 2 


3 


4 
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!!o\v did you u&i/ytho innovation? (Check all that apply* ) 

■\. TYPE OF PROGRAM 
D Pfeservice 



C Postsoeondary staff 
dev elooment 



□ 



Post secondary el assroon 



D Secondary inservice 
0 Secondary classroom 
O Other 



B . VOCATIONAL SERVICE AREA 

D Trade & industrial 
education 

D health occupations 
educaf ion 

□ Industrial arts education 

□ Technical education 



D Business & office 
education 



□ 
□ 
□ 
□ 



Distributive education 
Agi iculture education 
Home economics education 
Other 



10. Characterize your use of the innovation by circling' one 
number on each of the following scales . 



Frequent 



Scanned 
brie fly 



routine 

\ r>o \ for a 
sh^rt per i » >d 

Pr irary 

Alono 



i 



2 
2 

V 



3 
3 

3 
3 

3 



4 

4 
4 

4 
4 
4 



5 
■5 
5 



Seldori 

Studied 
intensively 

Creative 

Used for a 
long period 

Secondary 

With others 
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11. c^^r.n** f»ri|S^ use of t>>e, innovation by your organization by 
circlina one number on each of the followino scales. 

4 5 Many 

4 "5 Varied 

4 5 Difficult 

Col laborating 
4 * with others 

4 5 Separate * 

4 f? Creat i ve 

4 ^ Bonq-ranae 



Few • 12 3 

Uniform ] ? 3 

Smooth V 2^3 
Wo rk i no 

alone 1 ? 3 

Inteorated 1 ? 3 

Pontine 1 ? 3 

^hort-ranae 3 ? 3 



12. Record the amount of classroom time (in person-days/ P hours 
= 1 day) you spent actually bsino different sections of jthe 
innovation duririo the past year. 

i 





Ja n . 1 $ - , m 
M arch 31 


April. 1- - 
June 30 


Jfily I - 
Sept. 30 


Oct. 1 - 
Pec. 31 


TOTAL 
PFPSOF-DAYS 


Section T 
c ection TI 

• 

• 

• 

Sect ion ^ 




/ 
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13. T'Vnt 1 f y tbre^ Pionificant developments in your professional 
lifo that havp occurred a;» a result of usino the innovation. 



14. Mow puch has the innovation contributed to your persona] ^ 
arowfh on each of the followina irrensions? 





None 


A little 


Some ~*A 


Lot 


A Great Deal 


Added new Vnowledqe 


] 


2 


3 


4 


5 


Improved performance 


1 ' 


2 


3 


4 


5 1 


Influenced attitudes 


] 


2 


3 


4 


5 



15. Identify three significant contributions to your 

organization that have occurred "as a result of using the 
innovation. 



1 . 
2. 
3- 



16. !'ov much has the innovation contributed to oraani zat iona 1 
chanae on each of the followina dimensions? 

Fone A Iitt]e Fame A lot A Great Peal 



Fave' 1 t lVe ^ 

r^ved doJ lars 

Improved programs for 
c I ients 

Influenced policy 

Chanaed craani national 
structure 

Improved practices 
of staff 

T. 'Proved research 



2 
2 

2 

2 



2 
2 



3 
3 

3 
3 



3 
3 



4 
4 

4 
4 



4 
4 



5 

5 

5 
5 



5 
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N 

Based on your experience with the (innovation name) how 
satisfied are you with its overall performance? (Circle 
one ) 



1 Very^ dissatisfied . 
2. Somewhat dissatisfied 
3 ♦ Undecided/neutral 



4 . Some wh at satisfied 

5. Very satisfied 



Identify three significant long-range contributions to 
society that result fron this innovation* 

1. _ 

2. 

3. 
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APPENDIX 'B 
INTERVIEW GUIDE FOR INNOVATION IMPACT 
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INTERVIEW GUIDE FOR INNOVATION IMPACT 

» 

The following core ciuestions represent a less structured 
approach to iirpact assessment. iT-iey would be -asked of developers 
and users i.n a Qualitative interview. Other questions, based on 
responses to the initial auestion, would follow in each criterion 
category. 



CRITERIA 



COKE QUESTIONS FOR 
DEVELOPERS 



CORE QUESTIONS FOR 
USERS 



Systematically 
conducted 



High quality 



User-oriented 



Multiple 

communication 

channels 



Widespread 
dissemination 



1. How would you des- 
cribe the process 
followed in 'de- 
veloping the 
innovation? 

V* 

2 . How would you 
characterize the 
quality of the 
innovation? 

3 . How was the in- 
novation oriented 
to users? 



4. What dissemi- 
nation activities 
were conducted? 



5. Bow many C9pies of 
the innovation 
were disseminated 
by type of role, 
type of organi- 
zation, and geo- 
graphic location? 



1. How would you des- 
cribe the process 
followed in de- 
veloping the 
innovation? 

2. How would you 
characterize the 
quality of the * 
innovation? 

3 . How were* you in- 
volved in developing 
and implementing 
the innovation? 

4 . How did you re- 
ceive information 

, about this inno- 
vation? 

5. How many copies of 
the innovation did 
you receive? What 
were the roles, 
organizations and. 
location of users? 
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CRITERIA 



Strategic * 
dissemination 



Selective 
implementation 



Cost * 
feasibility 



Support 
systems 



Multiple 
' patterns 
of use 



Time^on task 



Integrated use 



CORE QUESTIONS FOR 
DEVELOPERS 



CORE QUESTIONS FOR 
USERS 



6. How was dissemi- 
nation individ- 
ualized for 
different types 
of usefs and 
sites? 

7. * What implemen- 

tation strategies 
were used to help 
each site ad v opt 
the innovation? ' 

8. ~What are the ■ * 

total and unit 
costs for imp'le-* 
menting the innova- 
tion? 

9-> What types of 
support were 
available for 
• implementing the 
innovation? 

10. How was the inno- 
vation used with 
different types of 
users? 

11. How inuch time was 
spent actually < - 
using the inno- 
vation in each 
setting? 

12 > How was the inno- 
vation integrated 
into ongoing 
programs and * 
practices? 



6. Describe .start-up 
• activities you 
participated in to 
leSrn how 'to use. tTie 
innovation. 



What factor^ unique 
to this s^te and to 
the people involved * 
affected imple- 
mentation?',- • 



( 



8.. What *have. been your ^ 
costs for. implement- 
ing the innovation* 
including in-.kind 
contributions? 

9. What type of 
support did you 
personally rebeive 
to' implement the ' 
innovation? 

10. How did you use ^ 
the innovation? 



11. How much time have 
you spent actually 
using this inno- 
vation? - 



12. How have you in- 
tegrated the inno- 
vation into your 
ongoing programs . 
and practices? 
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CRITERIA 



CORE QUESTIONS FOR 
DEVELOPERS 



CORE QUESTIONS FOR 
USERS 



User » 

satisfaction 



Individual 
growth 



- Organizational 
change 



Societal 
effects 



13 • How would you 

characterize 
users' < overall » 
reaction to the 
a nno.va t ion ? Wha t 
did they like. % 1} 
best? t What did / 
they like least? 

14 . How has the inno- 

vation con- 
tributed to the 
individual growth 
of users? 

15. How has the inno- 
vation contributed 
to organizational 
changes? 



16. How would you 

summarize the- most 
important long- 
term contributions 
of this innovation 
to improving 
- society? 



13. How would you de-' 
scribe your overall 
reaction to the 
innovation? what 
did you like best? 

* .*»W$iat* did you like 
'least? ■ 



15, 



16, 



14.*>How has partici- 
pating in the 
innovation affec- 
ted you perspnally? 



How has partici- 
pating in the inno- 
vation affected 
your program?- Your 
school? Your school 
system? 

How would you 
summarize the most 
important long- 
.term contributions 
of this innovation 
to improving 
society? 



95 



9 

ERLC 



101 



REFERENCES 



Adams, ha* A. C ase Studies of Research and Deve lopment Product 
Impact. Coiuwbus, OH: National Center for Research m 
Vocational Education,' The Ohio State University^ 1980. 

Adan'S, Kay; MacKay, brenda ; and Patton, Michael Q. Does -. 

Perfor mance-based Teacher Education Work?: Case Studies ot 
arTPriFTu rriculum for Vocational Teacher Education . 
Colunmus~oTTl NaTIonal Center for Research in Vocational 
Education, The OhKO State University, 1981. 

i ** 

Anderson, Jean, and hull, William L. • Can Research Help Men tally 
nandicapp ed Learners Explore Careers ? Columbus, OH: 
national Center for Research in Vocational Education, The 
Onio s£ate University, 198.1. , 

Benaan !>.,' and Paul y, E. Federal Programs Supp orting Educational 
Caange, Vol. 2: fac tors Affecting Change Ag ent Projects. 
Santa Monica, CA: Rahd Corporation, 1975." 

berman, Paul, and McLaughlin, Milbrey William. Federal Programs; 

supporting Educational Change, Vo l. 8: Implementing and 
^„ sustainin g Innovations . -Santa Monica, CA: Rand, May 1978., 

— ■ — ^ ^ w 

Bloom 3. S., ed. Taxonomy of Educational Objectives: Handbook • 

X , cognitive Doma in. New York: David McKay Co., 1956. 

Bragg, Debra D. Special Needs Impact Ev aluation: A Foll° w - u P 
"Study of the~Us e~of Bridges to Employme nt by CETA Staff. 
Columbus,' OH: National Center for Research m Vocational 
Education, The Ohio State University, April 1981. 

Bragg, Debra 0. Use and Effects of National Center for Resear ch 
-in Vocational Education Resources and Con ferences in 

Minnesota . Columbus, OH:- National Center for Research in 

Vocationa l Education, The Ohio State University, November 
1981. • • - " ' " 

Bragg, Debra*D.; ana Hull, William L. A Case Study of the 

"« vocatio nal Education Special Educ ation Project in Michigan. 

Columbu-s, oti ; National Center for Research in vocational 

Education, The Ohio State University, August 1981. 

Bragg, Debra D-; Hull, William L. ; and Adams, Kay A. The Impact 

oi Research an d Development pn Vocational Education for 

Handicapped Learners . Columbus, OH: National Center for 

Re search in Vocation al- Education, The Ohio State University, 
1981. 



97 



Brickell, Henry M. , and Paul, Regina H. Promoting and Achieving 
Sex Equity in , Vocational Education . Columbus, OH: National 
Center for Research in Vocational Education, The Ohio State 
University, 1979/* 

Brickell, Henry M. , and Paul, Regina H. Impact on Improving 
Evaluation in Vocational Education: A Study "of an " 
Evaluation Handbook on Follow-up Studies and Special Needs * 
Columbus, OH: National Center for Research in Vocational 
Education, The Ohio State University, 1980. 

Budke, Wesley B. State Vocational Education Program Improve - 
ment—Its Emphases r and Trends * presentation at the 
American Vocational Association Annual Conference, S£ . 
Louis, Missouri, 5 December 1982. •* 

4 

Carroll County Public Schools . Employability Skills for 
Special Needs Students * ^Westminister , MD: Carroll 
County Public Schools, 1978. ' 

4 

Center fQr Studies in Vocational Education, Employability Skills * 
TallahasseV FL: Florida* State University, 1977. 

Central Michigan University. Vocational Education Special 
* Education Project . Mt. Pleasant, MI: Central Michigan 
University, 1974. 

Central Michigan University. Vocational Education Special 

Education Project, II * Mt*. Pleasant, MI: Central Michigan 
University, 1978. ' " 

Chin,, Robert, and Benne, Kenneth D. "General Strategies for 
Effecting Changes in Human Systems." In The Planning of 
Cttange, 2nd ed . , edited by Warren G. Benni^ ; Kenneth • D. 
Benne; and Robert Chin. New York: Holt, Rinehart and 
/Winston, 1969. 

Clark, Dayid L., and Guba, Egon G. An Examination of Potential" 
Change Roles in Education . Paper presented at the 
Symposium oh Innovation in Planning School Curricula , 
Nirlie House, Virginia, 1965. • 

Committee on Vocational Education Reserarch and Development. 
Assessing- Vocational Education Research and Development . 
» Washington/ DC: National Academy of Sciences, 1976. 

Cotrell, Calvin J.; Chase, Shirley A.; and Molnar, Marilyn J. 
Model Curriculum for Vocational and Technical Teadher 
Education: Report No. 5 / Columbus, OH: National Center 
for Research in Vocational Education, The Ohio State 
University, 1972. • 



Crandall, David P.; Bau£hner/ Joyce Ellyn; Loucks, Susan F. ; and 
Schmidt, William H. Models of the School Improvement 
Process: Factors Contributing to Success ~ Paper presented 
at the American Educational Research Association Annual 
Meeting, March ^1982. 

Crowther/ F. "Factors affecting the rate of adoption of the 
1971 Alberta social studies curriculum for elementary 
students." Master's Thesis, University of Alberta, 19 72. 

Degner, David, ed. "Two Large-Scale Studies: Dissemination ^ and 
School Improvement." In Research and Educational Practice 
I mprovement Notes . San Francisco, CA: . Far West Laboratory 
April 1982. 

Development Associates, Inc. An Evaluation of Vocational 
Exemplary Projects: Part D, Vocational Education Act 
Amendments of 1968 . Washington, DC: Development 
Associates, Inc., March 1975. 

Dittenhafer, Clarence A., and Lewis, James P. Guidelines for 
Establishing Career Resource Centers * Harrisburg, PA: 
Pennsylvania Department of Education, 1976. 

Dissemination Analysis Group. Educational Dissemination in 
Relation to Public Elementary and Secondary Schools . 
Washington, DC: Department of Health, Education, and 
Welfare, 1977. 

Downs, Edward. Inside Bureaucracy . Boston, MA: Little Brown, 
1967. " 

fisher, C.W. ; Filby, N.V. ; Marbiave, R.S.; Cohen, L.S.; 
Dishaw, M.M.; Moore, J."E. ; and Berliner, D.C. 
Teaching Behaviors, Academic Learning Time and Student 
Achievement: Final Report , San Francisco: Far West 
Laboratory, 1978. 

Fitch, Brian. The Alliance for Career and Vocational Education 
"Columbus, OH: National Center for Research in Vocational 
Education, The Ohio State University, 1973. 

Florida Division of Vocational Education. Vital Informatioh 
for Education and Work . Tallahassee, FL: Florida State 
University, 1970. 

Francha^, Stephen J*, and Spirer, Janet E. Evaluation Handbook : 
Guidelines and Practices for Follow-up Studies' of Former 
Vocational Students" ! Columbus, OH: The National Center 
for Reserach in Vocational Education, The Ohio State 
University, 1978. 



Fullan/ M. , and Pomfret A. "Research on 'Curriculum and 
.Instruction Implementation . V Review of Educational 
Research '47 (1977): 335-397. r 

Gross,, N.-; Giaquinta, T. ; and Bernstein, M., Implementing © 
Organizational Innovation: A Sociological Analysis of 
Planned Educational Change . New York : Basic Books , 1971. 

Hall, Gene E. ; Loucks, Susan F^; Ruthelord, W.L. ; and Newlon, 

B . S . " Levels of Use of the Innovational : A Framework for 

Analysing Innovation Adoption." The Journal of Teacher 
. Education 29, (1975): 52-56. 

Hall,. Gene E :/ and Loucks, Susan F. M A Developmental Model fot 
Determining Whether the Treatment Is Actually Implemented." 
American Education Research Journal 14 , (1977): 263-276. 

Hamilton, James B.; Norton, Robert E. ; Fardlg, Glen E. ; 

- Harrington, Lois Gj? and # Quinn, Karen M. Performajice-based 
Professional Education Curricula: Vols. I and II . 
. Columbus , OH: [ National Center for* Research in Vocational 
EducatiQn, The Ohio State University, 1977. 

> 

Harrow, A. J. A Taxonomy' of the Psychomotor Domain . New York: 
* -David, McKay Co., r972. 

* 

Havelock, Ronald. Planning ::or Innovation through the 

Utilization of Scientific Knowledge . Ann Arbor, MI: . 
Institute for Social Research, 1971. 

Havelock ^tfonald G., and Lindquist "Jack . "A Conceptual 4 

. v Framework for Increasing "the Impact*" In Increasing the 

Impact , 'edited by Jaclc Lindquisi.* Battle Creek, MI: W.K. 
Kellogg Foundation, 3 980. 

Flood, ' P. and Blackwell, L. The Educational Information Market 
Study, Vol* -I and II . San Francisco: Far jWest Laboratory 
for Educational Research and Development, 19.76. 

— \ <? ' 

Hull, William L. Innovations Evaluation Guide: An Evaluation 

Tool for Innovation Consumers in Vocational-Technical 

Education . • Columbus, OH: National C&nt^r / for Research in 

Vocational Education, The Ohio State Univjeirsity, 1971. 

Hull, William L. Improving Vocational Education through 

Research . Paper presented at the Mid-Western* Educational 
Research Association Annual Conference,, Toledo, Ohio, % 
17 October 1980. - s " 

Hull, William L. Assessing the Impact of Three Research and 
Development Products in Pennsylvania . % Columbus, OFG • 
National Center for Research in Vocational Education, The 
Ohio State University, February 19Bla. 



100 



Hull, William L. Assessing the Impact of Two Research and 

Development Protlucts i n Florida * Columbus-, OH: National * 

Center for Research in Vocational Education, The Ohio Stajte 
University, October 1981b* 0 

Hiill, William L. , and Bragg, Debra D. Providing Vocational 
-Skills to Handicapped Students: ■ An Impact 'Study bf 
Teacher-developed Qurriculum .Guide s in .Maryland . Columbus , 
OH: National Center for Research ~in " m Vocational Education, 
The Ohio State University^, August 1981. * " 

Mull, William L. , cind Hassan S. The Impact of the National^ 
^ Academy's In-residence Program, 1978-1982 . Columbus, OH: 
National" Center for Research in Vocational Education*, 1 The 
" Ohio State University, July 1982. ' 

Illinois State* Board of Education^ Dissemination and *> 

Utilization of Vocational Education Program Improvement 
Products! . Springfield , 0 It: Illinois • State^ Board of 

Education, June 1980. . . , 1- 

* tt 

\ r 

Illinois State Board of Education. Impact Assessmerit^Design . ^ 
Assessing the Impact of Vocational Education Research and 
Development on Vocational. Education Project. Springfield, : 
Illinois State Board of Education, June 1982. 

Kiesler, Sara B. , and Turner, Charles F. / eds. Fundamental 
Research arid the P roce ss of Education . -Washington, DC: 
National * Academy of "Sciences , 1977. 

Kim, Jin Eun. "Federal Policy for and Impact Studies af :R&D 

Projects in Vocational Education." Educational Evaluation 
' ' and Policy Analysis 4 '(1982): 421-437'. - 

Komaski* Kenneth, Criteria* for Reviewing Education Produces T 
Education Products Information Exchange, 1978. 
* * * 

Kotler, Philip. "The Five C's: Course, Change Agency, Change 
Target, Channel, and Change Strategy." In Creating Social 
Change , edited *by Gerald Zaltman, Philip Kotler, and Ira 
Kaufman. New York: ' 'Holt, Rinehart .anc^ Winston . 
1972. v ' - 

Kr^thwohl, D. R. ; Bloomy B. S. ; and ^Masia. Taxonomy of 

HScXicational Objectives, Handbook II: Affective 'Domain . Me w 
York: David McKay Co., 1964. • t 

Krause, Scot N. , and Adams, Kay A. Product Review Exchange : 
Revised* Guidelines . Columbus, OH: National Center for 

* Research in Vocational Education, The Ohio State University, 
' 1982. * 



101 



Louis, Karen Seashore; Rosenblum, Slleila; and Moli*tor, Jame3 A. 
Linking R& D w ith Schools/ «Strategies #i for* Knowledge Use", and 
School Improvement ; A Summary, . Washington , DC * Abt 
Associates Inc., July 1981. < , ■» 

McDonald, !B '. , and Walker, R. , eds. Innovation Eval ua tion) 
Research, and the Problem of Control . Norwich, OK: 
SAFARI-PROJECT,. University of East Anglia*, 1.974. 

Modlsette, Laura, and Bonnett, Deborah G. Ca n Research Iniprove 
Career Guidance Programs in Rural Schools ? Columbus , OH : 
National Center for Research in^ Vocational Education, The 
.Ohio 'State University, 1981. 

ONatjLonal Cente^r for Research in Vocational Education. Career / 



Planning Support Sys 
State Universty, 19 



arch 
st|em. 
1Z7~ 



Columbus,- OH: NCRVE, The Ohio 



National Center for Research in Vocational Educa-tion. Rural? 
• America Guidanc e Series . Columbus, OH: -NCRVE, The Ohio" 
State University, 1977. * " 

National Center for Research in Vocational Education. Tentative 
Product Selec t ion 'Criteria . Columbus, OH: * NCRVE, The Ohio 
"State University, October 1$79* - 

National Ce'nter for Research in \7bcational Education. The * 

Distribution of Vocational Education Resea rc h and Exemplary 
Products . Colutnbus, OH: NCRVE, The Onio State University, ^ 
September 1930: • 



New York State Education Department. . Handbook of Aduli: Occu 
pational Education . Albany, . NY: 
Department,, 1977. 



New- York State Education 

V 



Orth, Molly N., and Russfell, Jill - Curric u lum Development ■ 

Needs for Vocational Education: New and Ch an ging Occupa - 
tions . Columbus, OH: ,NCRV£, The Ohip State University, 
' 1980. * ' c • < 

Report to the Congress: What Is the Role of Federal Assistance 
for Vo ca tional Education? Washington, * D.C.: Comptroller 
General "of the United States, 31 December 1974. (GAO 
Report) - : y 

Rogers, Everett /M. Diffusion of Innovations . New York: The 
Free Press, 1962. ' 

Rosenberg, Marc J. '"The ABCs 4 of ISD: Instructional Systems 
, 6esign." Training arid Development Journal 36 (1982-): 
* "44-50. ' • ' 



9 

ERIC 



•102 



/ 1 "1 
■ i 



Rothman, Jack* Planning and Organizing for Social Change . New 
York: Columbia University Press, 1974. 

Russell, Jill F. "Fmerging Occupations: Not Every Acorn Becomes 
An Oak", Occupational* Outlook Quarterly , Fall, 1982, pp. 
12-13. 

Sieber, Sam D. "Organizational Influence on Innovative Roles." 
In Knowledge Production and Utilization in Edytcationa* 
^AdmTn Tstration , edited by T.L. Eidell and J.M. Ketchel . 

Fug"ene, OR: Center for Study of Educational Administration, 
University of Oregon, 1968. 

>Sieber, S. "Organizational Influences on Inn<pv^tive /Roles . " In 
Knowledge Utilization in Public Education: <» Incentives and 
Di scentives in Improving Schopls:' Using~ What We Know~ T 
edited by R.Lehming, c*nd N. /Kane . > Sage Publication, 
Beverly Hills, 1981. 'A 



Stallings, J. A., 'and Kaskowitz, D. H. Follow Thro u gh Classroom 

* Observation Evaluation: Executive Summary . Menlo Park, CA: 
Stanford Research Institute, 1974. 

Suppes, Patrick, ed. Impact of Research on Education: Some Case 
Studies . Washington, DC: National Academy of Education, 
1978. ~ . , 

Thornfiike, Edward L. The?Psychology of Learning . New York: . 
Teachers College Press, Columbia University, 1913. 

Weick, K. "Educational organizations as loosely coupled 

systems." Administrative Scienpe Quarterly 21 (1976): 
1-9. ~~ 

"winkfield, Patricia Worthy; Storkr-Whitson, Karin; and Ripple, 

Gary. Bridges to Employment' . Columbus, OH: The National 
Center for ReseracJ^in Vocational' Educatioin, The Ohio 
State University, HL980. j 

Yin, Robert K. ; Heald, Karen A.; and Vogel, Mary E. Tinkering 
with the System . Lexington, MA: Lexington Books, 1977. 

Zaltman, Gerald, and Duncan, Robert. Strategies for Planned 
Change. New York: John Wiley & Sons, Inc., 1977. 

Zaltman, Gerald. "Organizational Settings and Innovation Use.^' 
In Increasing -the Impact , edited by Jack Lindquist. Battle 

* Creek, MI: W.K. Kellogg Foundation, 1980. 



ERLC 



103 % 

1"5 



